diff --git a/docs/unraid-os/using-unraid-to/run-docker-containers/run-command-line-tools-in-containers.mdx b/docs/unraid-os/using-unraid-to/run-docker-containers/run-command-line-tools-in-containers.mdx new file mode 100644 index 00000000000..b37beef1ee3 --- /dev/null +++ b/docs/unraid-os/using-unraid-to/run-docker-containers/run-command-line-tools-in-containers.mdx @@ -0,0 +1,304 @@ +--- +sidebar_position: 4 +sidebar_label: Run command-line tools in containers +--- + +# Run command-line tools in containers + +Some command-line tools, such as `ffmpeg`, `mediainfo`, and `exiftool`, are not part of Unraid. In the past, many people added them with NerdTools, NerdPack, or `un-get`. Those methods are no longer a good choice. This page shows how to run the same tools in short-lived Docker containers instead. + +A short-lived container starts when you run a command and is removed when the command ends. You do not install anything on Unraid, and you do not have to maintain it. + +## Why not install packages on Unraid + +- **NerdTools and NerdPack are deprecated.** For example, the [data recovery guide](../../troubleshooting/common-issues/data-recovery.mdx) states that "Nerd Tools and NerdPack are both deprecated." +- **`un-get` is archived.** The author of [`un-get`](https://github.com/ich777/un-get) writes in the README: "I would always recommend that you use Docker containers, LXC containers or VMs instead of running it on bare metal." +- **Missing libraries.** Unraid runs from RAM and ships a small set of packages. A tool that you add to it may need libraries that Unraid does not include, and it may fail when you run it. A container image includes the libraries that the tool needs. +- **Nothing is left behind.** Packages that you add to the host must be installed again at every boot. A container image is stored once and is easy to remove. + +## Before you start + +Check whether Unraid already includes the tool. These tools are part of a standard Unraid 7.3.4 installation, so you do not need a container for them: `rsync`, `htop`, `iperf3`, `nano`, `mc` (Midnight Commander), `curl`, `wget`, `dig`, `lsof`, `smartctl`, and `bsdtar`. + +`bsdtar` comes from the `libarchive` package and reads many archive types. See [Archives and RAR files](#archives-and-rar-files) before you install a separate archive tool. + +These tools are **not** part of Unraid 7.3.4: `ffmpeg`, `mediainfo`, `7z`, `unrar`, `exiftool`, `tcpdump`, `ncdu`, `nmap`, `tmux`, and `screen`. + +:::note + +`tmux` and `screen` keep a session alive after you disconnect. A short-lived container ends when its command ends, so they do not fit this method. Use the web terminal or a long-lived container instead. + +::: + +The commands on this page run in the Unraid terminal. For more information, see [Accessing the terminal](../../system-administration/advanced-tools/command-line-interface.mdx#accessing-the-terminal). + +## How a short-lived container works + +This example reads the length of a video with `ffprobe`: + +```bash +cd /mnt/user/media/videos +docker run --rm --user 99:100 --entrypoint ffprobe \ + -v "$PWD":/work:ro -w /work \ + linuxserver/ffmpeg -v error -show_entries format=duration -of csv=p=0 movie.mp4 +``` + +Each option has a purpose: + +| Option | Purpose | +| --- | --- | +| `--rm` | Removes the container when the command ends. See [What `--rm` does](#what-rm-does). | +| `--user 99:100` | Runs the tool as `nobody:users`, the owner of most Unraid shares. New files get the same owner as other files on your shares. | +| `-v "$PWD":/work:ro` | Makes the current folder visible inside the container as `/work`. `:ro` means read-only, which is safe for tools that only read. Remove `:ro` for tools that write files. | +| `-w /work` | Starts the command in `/work`, so you can use short file names. | +| `--entrypoint ffprobe` | Starts `ffprobe` instead of the default program of the image. | + +The tool can only see the folder you mount. Paths such as `../other-folder` do not work inside the container. + +The first run downloads the image and stores it in `docker.img`. For more information, see [Overview](./overview.mdx). + +## Choose the tools and images + +The table shows an example for each tool. The images come from their maintainers on Docker Hub. + +| Tool | Image | Example | +| --- | --- | --- | +| `ffmpeg`, `ffprobe` | `linuxserver/ffmpeg` | `docker run --rm --user 99:100 --entrypoint ffmpeg -v "$PWD":/work -w /work linuxserver/ffmpeg -i in.mp4 -vf scale=-2:480 out.mp4` | +| `mediainfo` | `cli-tools` (built below) | `docker run --rm --user 99:100 -v "$PWD":/work:ro -w /work cli-tools mediainfo movie.mp4` | +| `7z` (`.7z`, `.zip`, encrypted 7z) | `cli-tools` | `docker run --rm --user 99:100 -v "$PWD":/work -w /work cli-tools 7zz a backup.7z folder` | +| `unrar` (only if `bsdtar` cannot read the file) | `rar-tools` (built below) | `docker run --rm --user 99:100 -v "$PWD":/work -w /work rar-tools 7z x archive.rar` | +| `exiftool` | `cli-tools` | `docker run --rm --user 99:100 -v "$PWD":/work -w /work cli-tools exiftool photo.jpg` | +| `ncdu` | `cli-tools` | `docker run --rm -it --user 99:100 -v "$PWD":/work:ro -w /work cli-tools ncdu` | +| `tcpdump` | `cli-tools` | `docker run --rm --net=host cli-tools tcpdump -n -i br0 -c 20` | +| `nmap` | `instrumentisto/nmap` | `docker run --rm --net=host instrumentisto/nmap -Pn -p 22,80 192.168.1.1` | + +The `linuxserver/ffmpeg` project describes its image as "meant to be run ephemerally from the command line." This is the intended use. The `cli-tools` image is described in the next section. + +Source projects: [`linuxserver/ffmpeg`](https://github.com/linuxserver/docker-ffmpeg), [`instrumentisto/nmap`](https://github.com/instrumentisto/nmap-docker-image), and [Alpine Linux](https://hub.docker.com/_/alpine) (the base of `cli-tools`). + +### Archives and RAR files + +Try the `bsdtar` that is already in Unraid first. It needs no container: + +```bash +bsdtar -tf archive.rar # list the files +bsdtar -xf archive.rar -C /mnt/user/media/unpacked # extract into an existing folder +chown -R nobody:users /mnt/user/media/unpacked # give the files the usual share owner +``` + +When you run `bsdtar` as root, which is the default in the Unraid terminal, it keeps the owner and permissions that are stored in the archive. The extracted files can then belong to a user ID that does not exist on Unraid, and `nobody:users` (`99:100`) may not be able to write to them. The `chown` command fixes the owner. The `--no-same-owner` option does not help here: it gives the files to root instead. + +`bsdtar` also reads `.zip`, `.7z`, and `.iso` files, and it can create them. The tests behind this page found these results for RAR files, using the same `libarchive` version (3.8.10) that Unraid 7.3.4 includes: + +| RAR file | `bsdtar` in Unraid | Fallback | +| --- | --- | --- | +| RAR5 or RAR4, list and extract | Works | Not needed | +| Solid archive (`rar -s`) | Works | Not needed | +| Password-protected | Fails: `Encryption is not supported` | `rar-tools` | +| Multi-volume set (`.part1.rar`, `.part2.rar`) | Fails with `Too small block encountered` | `rar-tools` | + +The `libarchive` project documents RAR support as limited. Its [README](https://github.com/libarchive/libarchive/blob/master/README.md) lists "RAR and RAR 5.0 archives (with some limitations due to RAR's proprietary status)". The multi-volume failure is a known regression in 3.8.10 ([issue 3683](https://github.com/libarchive/libarchive/issues/3683)). It was fixed after that release ([pull request 3686](https://github.com/libarchive/libarchive/pull/3686)), so a later Unraid release may not need the fallback. + +For the cases that `bsdtar` cannot read, build the `rar-tools` image in the next section. It uses the Debian `7zip` and `7zip-rar` packages. `7zip-rar` is in the `non-free` area of Debian because the RAR code uses the unRAR licence. The 7-Zip package in Alpine, used by `cli-tools`, leaves out RAR support for the same reason. + +### Build the images + +1. Create a folder with two files, for example on a share: + + ```bash + mkdir -p /mnt/user/appdata/cli-tools + cd /mnt/user/appdata/cli-tools + ``` + + ```dockerfile title="Dockerfile" + FROM alpine:3 + RUN apk add --no-cache mediainfo 7zip exiftool tcpdump ncdu + ``` + + ```dockerfile title="Dockerfile.rar" + FROM debian:trixie-slim + RUN sed -i 's/^Components: main$/Components: main contrib non-free/' /etc/apt/sources.list.d/debian.sources \ + && apt-get update \ + && apt-get install -y --no-install-recommends 7zip 7zip-rar \ + && rm -rf /var/lib/apt/lists/* + ``` + +2. Build the images. Build as root, which is the default in the Unraid terminal: + + ```bash + docker build -t cli-tools . + docker build -f Dockerfile.rar -t rar-tools . + ``` + + `cli-tools` is about 125 MB and `rar-tools` is about 128 MB. Build `rar-tools` only if you need it. + +:::info[Why build images?] + +No widely used image exists for `mediainfo` or `exiftool` alone. A small image that you build once also avoids a package download every time that you run a tool. + +::: + +### Network tools + +- `--net=host` lets the container see the network interfaces of Unraid. Without it, `tcpdump` only sees the interfaces of the container. +- Use `br0`, `bond0`, or `eth0` as the interface name, depending on your network settings. + +### Hardware transcoding with an Intel GPU + +Add `--device /dev/dri` to give `ffmpeg` access to the Intel GPU. You also need the hardware options of `ffmpeg`. For example, this command encodes with VAAPI: + +```bash +# Find the render device of the Intel GPU (vendor 0x8086) +for d in /sys/class/drm/renderD*; do + [ "$(cat "$d/device/vendor")" = 0x8086 ] && RENDER=/dev/dri/${d##*/} && break +done +echo "$RENDER" # for example /dev/dri/renderD128 + +docker run --rm --device /dev/dri --user 99:100 \ + --group-add "$(stat -c %g "$RENDER")" --entrypoint ffmpeg \ + -v "$PWD":/work -w /work linuxserver/ffmpeg \ + -init_hw_device vaapi=va:"$RENDER" -filter_hw_device va \ + -i in.mp4 -vf format=nv12,hwupload -c:v h264_vaapi out.mp4 +``` + +The device name is often `renderD128`, but it can differ when the server has more than one GPU. If `echo "$RENDER"` prints nothing, the server has no Intel GPU. + +`--group-add` gives the container the group that owns the GPU device, so a non-root user can open it. Without `--device /dev/dri`, the command fails with `No VA display found`. The `qsv` encoders of `ffmpeg` also work through this device. + +## What `--rm` does {/* #what-rm-does */} + +Every example above uses `--rm`. This section explains the difference between running with and without it. + +| | With `--rm` | Without `--rm` | +| --- | --- | --- | +| Container after the command ends | Removed | Stays, in the `Exited` state | +| Listed by `docker ps -a` | No | Yes | +| Files written inside the container | Lost | Kept in the container until you remove it | +| Exit code of `docker run` (`$?` in the shell) | Same as the command | Same as the command | +| Logs and stored exit code afterwards | Lost | Kept (`docker logs`, `docker inspect`) | +| Anonymous volumes | Removed with the container | Stay until you remove them | +| Named volumes | Stay | Stay | +| Files written to a bind mount (`-v folder:/work`) | Kept | Kept | +| Disk space in `docker.img`: image | Used once, by the first pull or build (about 125 MB for `cli-tools`) | Same | +| Disk space in `docker.img`: each run | No growth | The writable layer of each kept container | +| `--name` can be used again | Yes | No, until you remove the container | + +The key point is that results written to a bind mount stay in the folder on your server in both cases. Only files written inside the container are lost with `--rm`. + +### What stays behind without `--rm` + +A container that is not removed stays in the list of stopped containers. In this example, the command fails on purpose: + +```bash +docker run --name test1 alpine sh -c 'echo hello; echo oops >&2; echo result > /tmp/result.txt; exit 3' +``` + +The `docker run` command returns the exit status of the command, so `echo $?` prints `3` here, with or without `--rm`. You can then inspect the stopped container: + +```bash +docker ps -a --filter name=test1 # shows "Exited (3)" +docker logs test1 # shows the saved output +docker inspect test1 --format '{{.State.ExitCode}}' +docker diff test1 # lists files changed inside the container (shows /tmp/result.txt) +docker cp test1:/tmp/result.txt . # copies the file out of the container +docker start -a test1 # runs the same command again +docker rm test1 # removes it +``` + +A kept container also appears on the **Docker** tab of the %%WebGUI|web-gui%% in the stopped state. + +### Disk use and name conflicts + +- **Disk use.** The image is a one-time cost that `--rm` does not change. Each container without `--rm` also keeps its own writable layer, which is the per-run growth. Twenty runs that each wrote 10 MB inside the container kept about 200 MB. On Unraid, this space comes from `docker.img`, which can fill up. For more information about a full `docker.img`, see [Docker troubleshooting](../../troubleshooting/common-issues/docker-troubleshooting.mdx). +- **Names.** A container name must be unique. A second `docker run --name test1` fails with `Conflict. The container name "/test1" is already in use`. This also applies to `--rm`, while the first container is still running. +- **Volumes.** `--rm` removes anonymous volumes (for example `-v /data`) but not named volumes (for example `-v mydata:/data`). Without `--rm`, a plain `docker rm` leaves anonymous volumes behind. Use `docker rm -v` to remove them too. + +### Clean up + +To remove one container, use `docker rm `. To remove many, use a label so that you only remove your own containers: + +```bash +docker run --label cli-test=1 --name job1 alpine true +docker container prune --filter label=cli-test=1 +``` + +:::caution + +Do not run `docker container prune` without a filter. It removes **every** stopped container, including containers that you stopped on purpose on the **Docker** tab. You must then create them again. + +::: + +### Which one should you use? + +| Situation | Use | +| --- | --- | +| One-off commands such as `ffprobe`, `mediainfo`, `exiftool` | `--rm` | +| You need the logs or stored exit code of a failed run afterwards | A named container without `--rm` | +| You want to run the same configuration again with `docker start` | A named container without `--rm` | +| You want to copy a file out with `docker cp` | A named container without `--rm` | +| A long job that you check later | A named container without `--rm`, or `-d --name job` | +| A service that should keep running | `-d --restart unless-stopped`, or install it from Community Applications | + +## Traps and quirks + +Each item shows what happens, why, and the fix. "Observed" means that the behavior was seen in testing and that no document describes it. + +- **`--user` fails with `linuxserver` images.** The container prints `groupmod: Permission denied` and `s6-applyuidgid: fatal`. These images start as root to set up their user. The [LinuxServer.io documentation](https://docs.linuxserver.io/general/understanding-puid-and-pgid/) says: "Our images are not yet compatible with this, so we recommend continuing usage of PUID and PGID." Fix: skip the start-up script with `--entrypoint ffmpeg --user 99:100`. +- **`PUID` and `PGID` do not set the owner of `ffmpeg` output files.** The `linuxserver/ffmpeg` image does not use them. Its [README](https://github.com/linuxserver/docker-ffmpeg) says: "If an input file is detected we will run FFmpeg as that user/group so the output file will match its permissions." Observed: an output file got the same owner as the input file, whichever owner that was (a normal user, or `99:100`). Fix: `--entrypoint ffmpeg --user 99:100`. +- **`apk add` fails as a normal user.** The error is `Unable to open log: Permission denied`, because installing packages needs root. Observed. Fix: install the packages when you build the image, and run the tools with `--user`. +- **`../` paths do not work.** Only the current folder is mounted, so `../other` does not exist in the container. Observed. Fix: run the command from a parent folder, or add another `-v`. +- **`tcpdump` sees only the container network.** Observed: a container on the default network has only `eth0` and `lo`. Fix: `--net=host`. Capturing packets needs the `NET_RAW` capability. The [Docker documentation](https://docs.docker.com/engine/containers/run/) lists `NET_RAW` among the capabilities that are on by default, so you only see an error if you remove it, for example with `--cap-drop ALL`. +- **Intel GPU: `No VA display found`.** The container does not get devices unless you add them. The Docker documentation says that `--device` "Allows you to run devices inside the container without the `--privileged` flag.", and the [`linuxserver/ffmpeg` documentation](https://docs.linuxserver.io/images/docker-ffmpeg/) uses `--device=/dev/dri:/dev/dri`. Observed: with `--user 99:100`, the render device also needs its group. Fix: `--group-add "$(stat -c %g "$RENDER")"`, with `$RENDER` set as in [Hardware transcoding with an Intel GPU](#hardware-transcoding-with-an-intel-gpu). The Docker documentation describes `--group-add` as: "Add additional groups to run as". +- **Anonymous and named volumes.** The Docker documentation says: "Only volumes that are specified without a name are removed." removing the container this way is "similar to running `docker rm -v my-container`". Observed: a plain `docker rm` leaves anonymous volumes behind, and named volumes always stay. Fix: use `docker rm -v`, or prefer bind mounts. +- **Name conflicts.** A second `docker run --name job1` fails with `Conflict. The container name "/job1" is already in use`, while the first container exists. Observed. Fix: `docker rm job1`, or leave out `--name` when you use `--rm`. +- **`docker container prune` removes every stopped container.** The [Docker documentation](https://docs.docker.com/reference/cli/docker/container/prune/) says: "Removes all stopped containers." That includes containers that you stopped on purpose. Fix: give your own containers a label and use `--filter label=...`, or use `docker rm `. +- **Disk use in `docker.img`.** Without `--rm`, each run keeps its changes. The [Overview](./overview.mdx) page says that "All the data and programs for each container are stored in a single virtual disk image file called `docker.img`." Observed: 20 runs that each wrote 10 MB kept about 200 MB. Fix: use `--rm`, or remove kept containers. +- **Alpine `7zip` cannot open RAR files.** The error is `Cannot open the file as archive`. The [Alpine package recipe](https://raw.githubusercontent.com/alpinelinux/aports/master/main/7zip/APKBUILD) says "DISABLE_RAR: RAR codec is non-free". Debian puts the same code in a separate `non-free` package. Its [copyright file](https://metadata.ftp-master.debian.org/changelogs//non-free/7/7zip-rar/7zip-rar_25.00+ds-1+deb13u1_copyright) says: "RAR code is under the non-free "unRAR license restriction"". Fix: use `bsdtar`, or the `rar-tools` image. +- **`bsdtar` cannot read encrypted or multi-volume RAR files.** See [Archives and RAR files](#archives-and-rar-files). The `libarchive` [issue 1661](https://github.com/libarchive/libarchive/issues/1661) describes the missing encryption support, and [issue 3683](https://github.com/libarchive/libarchive/issues/3683) describes the multi-volume failure in 3.8.10. Fix: `rar-tools`. +- **`tmux` and `screen` sessions end with the container.** Observed: a detached `tmux` session started in one `docker run --rm` was gone in the next run (`error connecting to /tmp/tmux-0/default`). Fix: use the web terminal, or a long-running container with `docker exec`. + +## Make the commands feel native + +Typing long `docker run` lines is slow. Define shell functions so that you can type `ffprobe movie.mp4` as before. Save this as a file, for example `/boot/config/cli-tools.sh` (you can choose any file name on the boot device): + +```bash title="/boot/config/cli-tools.sh" +_tty() { [ -t 0 ] && [ -t 1 ] && echo -it || echo -i; } +ffmpeg() { docker run --rm $(_tty) --user 99:100 --entrypoint ffmpeg -v "$PWD":/work -w /work linuxserver/ffmpeg "$@"; } +ffprobe() { docker run --rm $(_tty) --user 99:100 --entrypoint ffprobe -v "$PWD":/work:ro -w /work linuxserver/ffmpeg "$@"; } +mediainfo() { docker run --rm --user 99:100 -v "$PWD":/work:ro -w /work cli-tools mediainfo "$@"; } +7z() { docker run --rm --user 99:100 -v "$PWD":/work -w /work cli-tools 7zz "$@"; } +unrar() { docker run --rm --user 99:100 -v "$PWD":/work -w /work rar-tools 7z "$@"; } +exiftool() { docker run --rm --user 99:100 -v "$PWD":/work -w /work cli-tools exiftool "$@"; } +ncdu() { docker run --rm $(_tty) --user 99:100 -v "$PWD":/work:ro -w /work cli-tools ncdu "$@"; } +tcpdump() { docker run --rm $(_tty) --net=host cli-tools tcpdump "$@"; } +``` + +Run `source /boot/config/cli-tools.sh` to use the functions in the current session. Each function mounts only the folder you are in. + +### Keep the functions after a reboot + +Unraid runs from RAM, so changes to the shell environment are lost at every reboot. Commands that must run at boot belong in the `go` file at `/boot/config/go`, as described in [Wake on LAN](../../system-administration/advanced-tools/wake-on-lan.mdx). A community answer on the Unraid forum gives the same advice for shell aliases: put the commands in a script, and call the script from the `go` file ([Bash aliases persisting after reboot](https://forums.unraid.net/topic/58849-bash-aliases-persisting-after-reboot/)). + +Add this line to `/boot/config/go` so that your functions are loaded at login. Open a new terminal session after the next reboot to use them: + +```bash +echo 'source /boot/config/cli-tools.sh' >> /root/.bash_profile +``` + +Also rebuild the `cli-tools` image if it is missing after you recreate `docker.img`. + +### Use the tools in scheduled scripts + +The [User Scripts plugin](./managing-and-customizing-containers.mdx#user-scripts-plugin) runs scripts on a schedule. Inside a script, call `docker run --rm` directly instead of the functions. For example, this script saves a media report for a folder: + +```bash +#!/bin/bash +docker run --rm --user 99:100 -v /mnt/user/media/videos:/work:ro cli-tools \ + sh -c 'mediainfo --Output=JSON /work/*.mp4' > /mnt/user/media/report.json +``` + +## Related pages + +- [Overview](./overview.mdx) +- [Managing & customizing containers](./managing-and-customizing-containers.mdx) +- [Command line interface](../../system-administration/advanced-tools/command-line-interface.mdx)