Skip to content

Warming the rclone cache at startup

After a reboot, your rclone VFS cache starts cold. The first time your media server browses the library, rclone has to fetch every directory listing from the S3 backend on demand, so the first browse or scan feels slow. Warming the directory listings once, right after the mount comes up, fixes that.

This is the boot-time companion to ZenLocalPoller: the poller keeps your view in sync while the system is running; warming the cache primes it once at startup.

What this warms — and what it deliberately doesn't

The recommended startup warm fetches directory metadata only, non-recursively, for the immediate top level of each mount. It warms the listings you browse, not every nested folder or media file. That is what makes browsing the ZenDRIVE library feel instant after a reboot while staying light on the backend — and it is the approach we run in production.

The ZenDRIVE catalog is large, so warming the whole tree at boot is heavy: it makes a big burst of backend requests and, on some setups, has been enough to stall the mount (stuck D-state reads). Keep the startup warm light:

  • Refresh directories only, never media files.
  • Use recursive=false and only the immediate top-level folders.
  • Use limited parallelism (a handful at a time, not hundreds).

If you want the whole tree cached instead

The top-level warm makes browsing fast but does not pre-cache the deep folders a full library scan walks into. If you specifically want a complete first scan to read entirely from cache, run the optional full deep prime once (a recursive vfs/refresh with --fast-list) — heavier on a library this size, so run it off-peak.

How is this different from ZenLocalPoller?

They solve two different moments. Run both.

Startup warm ZenLocalPoller
When Once, just after boot / mount start Continuously, while running
Trigger Boot / container start MQTT events for new content
Scope Top-level directory listings The specific path that changed
Goal Fast first browse after a reboot New content appears right away

How do I warm the directory metadata?

The idea is to fetch only the directory structure you browse. The immediate top-level listings are what make browsing feel instant; the files inside are streamed on demand and do not need pre-caching.

The pattern, per mount:

  1. Wait for the RC endpoint to be ready by polling rc/noop until it answers.

  2. Refresh the mount root, non-recursive:

    { "recursive": "false" }

  3. Discover the immediate child directories (top level only):

    find "$mount_path" -mindepth 1 -maxdepth 1 -type d

  4. Refresh each immediate directory, non-recursive, a few at a time:

    rclone rc vfs/refresh --rc-addr 127.0.0.1:PORT \
  dir="DIRECTORY_NAME" recursive=false

    The equivalent raw call is POST /vfs/refresh with body { "dir": "DIRECTORY_NAME", "recursive": "false" }.

That is the whole idea: root listing + one non-recursive refresh per top-level folder, per mount.

Skip folders that belong to another mount

If you split the library across mounts — for example a separate mount for an int (internal) subtree — skip that subfolder on the parent mount so you don't warm it twice. Let the dedicated mount warm its own root.

A worked example

If your library is presented through several rclone mounts (for example aggregated with mergerfs/unionfs), warm each mount on its own RC port. The values below are representative — use your own mount paths and RC ports:

Mount Path RC address
movies /mnt/remote/zendrive/movies 127.0.0.1:5572
television /mnt/remote/zendrive/television 127.0.0.1:5573
sports /mnt/remote/zendrive/sports 127.0.0.1:5574
courses /mnt/remote/zendrive/courses 127.0.0.1:5576

A small POSIX shell script that implements the non-recursive pattern:

#!/bin/sh
# Warm rclone VFS *directory* metadata at startup. Directories only, non-recursive.
set -eu

START_DELAY_SECONDS="${START_DELAY_SECONDS:-15}"
MAX_PARALLEL_REFRESHES="${MAX_PARALLEL_REFRESHES:-3}"   # keep this small
DIR_TIMEOUT_SECONDS="${DIR_TIMEOUT_SECONDS:-1800}"

# mount_name <TAB> path <TAB> rc_addr
MOUNTS="
movies  /mnt/remote/zendrive/movies 127.0.0.1:5572
television  /mnt/remote/zendrive/television 127.0.0.1:5573
sports  /mnt/remote/zendrive/sports 127.0.0.1:5574
courses /mnt/remote/zendrive/courses    127.0.0.1:5576
"

rc() {  # rc <addr> <path> [json]
  wget -q -O- --post-data="${3:-{}}" "http://$1/$2" >/dev/null 2>&1
}

wait_for_rc() {  # block until the RC endpoint answers rc/noop
  addr="$1"
  while ! rc "$addr" "rc/noop"; do sleep 2; done
}

refresh_dir() {  # non-recursive refresh of one immediate directory
  addr="$1"; dir="$2"
  timeout "$DIR_TIMEOUT_SECONDS" sh -c \
    "wget -q -O- --post-data='{\"dir\":\"$dir\",\"recursive\":\"false\"}' \
      'http://$addr/vfs/refresh' >/dev/null 2>&1" \
    && echo "ok   $addr $dir" || echo "FAIL $addr $dir"
}

sleep "$START_DELAY_SECONDS"

echo "$MOUNTS" | while IFS='    ' read -r name path addr; do
  [ -n "$name" ] || continue
  echo "mount=$name path=$path rc=$addr"
  wait_for_rc "$addr"

  # 1. warm the mount root (non-recursive)
  rc "$addr" "vfs/refresh" '{"recursive":"false"}'

  # 2. warm each immediate top-level directory, a few at a time
  running=0
  find "$path" -mindepth 1 -maxdepth 1 -type d | while read -r d; do
    refresh_dir "$addr" "$(basename "$d")" &
    running=$((running + 1))
    if [ "$running" -ge "$MAX_PARALLEL_REFRESHES" ]; then
      wait; running=0
    fi
  done
  wait
done

echo "status=all-done"

Run it once at boot — as a systemd oneshot, a small container, or a startup hook on the host that owns the mount. If you run it in a container, give it host networking so it can reach the RC ports on 127.0.0.1, and bind-mount the remote read-only so find can list the directories.

Tune parallelism carefully

Three concurrent refreshes is a safe default. Raising it makes more backend requests at once; only increase it if you have confirmed your backend and mount handle it without stalling. More parallelism is not faster if it triggers throttling or stalls.

How do I verify it worked?

  • Watch the log. Each mount should print its name, the directories it queues, and finish with status=all-done.
  • After it finishes, browse a top-level library in your media server — for example /movies and its immediate subfolders. The listing should appear quickly instead of stalling on the first open.

  • Confirm the mount is healthy, with no stuck reads:

    ps -eo pid,state,wchan:40,cmd | grep -E 'ffprobe|ffmpeg|cat .*mnt' | grep -v grep

    You should see no processes stuck in D (uninterruptible) state. Stuck reads are a sign the warmup is too heavy — back off the parallelism and keep it non-recursive.

Do I still need ZenLocalPoller?

Yes — they complement each other. The startup warm makes the first browse after a reboot fast; ZenLocalPoller keeps your mount in sync as new content arrives afterwards. See How live updates work for the live side.