scene-actor-extraction/docs/service-conversion.md

Conversion to service — a native idle-GPU worker

Status: design / proposal. Nothing here is built yet.

The idea

Turn the CLI tools into a turnkey batch worker that uses the machine's idle GPU: it analyses newly-added Jellyfin media when you're not using the computer (screen locked), and stops the instant you come back. It's an overnight job on your own Linux box.

No Docker. This runs on your own machine with your own drivers, so a container buys little and costs a lot: GPU passthrough (nvidia-container-toolkit, or /dev/kfd+/dev/dri+video group for ROCm) is the single most fragile part of a containerised setup, and it exists only because of the container. Natively, the GPU just works with the drivers you already have, and the media paths Jellyfin reports are just real paths — no re-mounting. So we ship a native installer instead of an image builder.

Two deliverables:

1. An installer — scripts/build_install.py. Detects your distro, ensures the GPU/build dependencies are present (via dnf/pacman), compiles scene_analyze for your GPU, and installs the binary + Python glue + two systemd user units under ~/.local.
2. A screen-lock gate — one of those systemd units watches logind lock/unlock and starts/stops the worker. Lock → analyse. Unlock → stop.

What already exists (reuse, don't rebuild)

The processing loop is already implemented — this is packaging, building, and lock-gating, not new pipeline logic.

Piece	Where	What it does
Analysis engine	build/scene_analyze	Video → face detect/align/embed → gallery match → result JSON
Backend selection	CMakeLists.txt (SAE_INFERENCE_BACKEND, SAE_GEMM_BACKEND)	ORT/TRT + ROCm/CUDA, chosen at build time
New-media queue	JRay plugin → GET /Plugins/JRay/Tasks/Pending	Backlog of items with no results yet
Worker loop	run_from_jellyfin.py --worker	Poll Pending → run scene_analyze → push results
Result push	PUT /Plugins/JRay/Items/{id}/Truth	Stores per-actor scene windows back in Jellyfin
Incremental gallery	make_jellyfin_gallery.py --merge	Embeds only cast not already in the gallery
Secrets loader	.env via sae_env.py	JELLYFIN_URL, JELLYFIN_API_KEY, TMDB_API_KEY

Installer config

One file. Build-time settings (fixed when we compile) vs. run-time settings (in the worker's .env, editable without recompiling).

# install.yaml — consumed by scripts/build_install.py

platform: nvidia            # nvidia | amd | cpu → picks the cmake backend
model:
  arcface: LVFace-B_Glint360K.onnx   # embedder compiled against; gallery MUST match
schedule:
  gallery_scan_interval: 24h         # incremental --merge cadence; 0 disables the scanner
prefix: ~/.local                     # install root (bin, share, systemd user units)

# runtime (written to the worker .env, not compiled in):
runtime:
  jellyfin_url: http://localhost:8096
  # JELLYFIN_API_KEY / TMDB_API_KEY are filled into .env by hand after install

Secrets never go in the repo or a build artifact — the installer writes a .env under the install prefix with blanks for the keys, and you fill them in once. sae_env.py already loads it.

Model ⇄ gallery coupling (guard, don't just document): embeddings from different recognition models aren't interchangeable. We compile against one embedder; the gallery must be built with the same one. Stamp the embedder name into gallery.json, and have the worker refuse to start if the gallery's embedder ≠ the configured model.arcface, rather than silently mismatching.

Dependencies via the system package manager

The heavy build/runtime deps (OpenCV, ffmpeg, the GPU stack) are best provided by the distro, not vendored. The installer ships a per-distro dependency list and either installs them or prints the exact command. Targets: Fedora (dnf) and Arch (pacman) first.

Dependency	Fedora (dnf)	Arch (pacman)
OpenCV	opencv-devel	opencv
ffmpeg	ffmpeg-free/ffmpeg (RPM Fusion)	ffmpeg
CMake / toolchain	cmake gcc-c++	cmake gcc
CUDA + TensorRT (nvidia)	NVIDIA CUDA repo + libnvinfer-*	cuda, tensorrt
ROCm (amd)	rocm-hip-sdk / rocblas-devel	rocm-hip-sdk, rocblas
ONNX Runtime	not packaged — installer fetches a pinned release tarball into the prefix	AUR onnxruntime (or same pinned-tarball fallback)

So the flow is: detect distro → check each package → install via the native manager (or print sudo dnf install … / sudo pacman -S …), with ONNX Runtime as the one known gap the installer fills itself (a pinned upstream release extracted under the install prefix, so it doesn't depend on a system package that may not exist). CUDA/ROCm being present is assumed — you already run a GPU desktop; the installer verifies and points you at the vendor repo if not.

What build_install.py does

build_install.py install.yaml
  │
  ├─ detect distro (dnf vs pacman) and platform from config
  ├─ ensure deps: install via manager, or print the exact command; fetch ONNX Runtime if needed
  ├─ cmake + build scene_analyze with the platform's backend flags:
  │     nvidia → -DSAE_INFERENCE_BACKEND=TRT -DSAE_GEMM_BACKEND=CUDA
  │     amd    → -DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=ROCM
  │     cpu    → -DSAE_INFERENCE_BACKEND=ORT  (CPU EP; slow, for smoke tests)
  ├─ install into <prefix>:
  │     bin/sae-scene-analyze           the compiled binary
  │     share/sae-worker/               Python glue + a venv (requests, etc.), models/
  │     share/sae-worker/.env           runtime config (keys blank, url from config)
  ├─ install systemd --user units:
  │     sae-worker.service              runs the worker + gallery-scan supervisor
  │     sae-lock-gate.service           watches logind lock/unlock, start/stops the worker
  └─ print next steps (edit .env, `systemctl --user enable --now sae-lock-gate`)

The worker service (supervisor)

sae-worker.service runs a small Python supervisor as its main process:

• starts the worker loop (run_from_jellyfin.py --worker) — the hot path,
• starts a gallery-scan timer — sleeps gallery_scan_interval, runs make_jellyfin_gallery.py --merge, repeats,
• exits cleanly on SIGTERM (see re-queue below).

The lock gate

sae-lock-gate.service runs a tiny watcher that subscribes to logind lock/unlock signals and drives the worker service:

screen locks   → systemctl --user start sae-worker.service
screen unlocks → systemctl --user stop  sae-worker.service   (SIGTERM)

Screen-lock is the only signal — deliberately. We don't also gate on GPU/CPU load, because our own worker is the load: a load threshold would form a feedback loop (worker starts → GPU spikes → threshold trips → worker stops → load drops → restart → …). Lock state is external to what the worker does, so it can't oscillate.

Signal source is desktop-dependent: logind Lock/Unlock (GNOME/KDE via loginctl/D-Bus) covers most setups; a swayidle/xss-lock hook is the fallback for wlroots/X-only compositors. The installer picks based on what's present.

On resume: hard stop + re-queue (it's free)

Stopping the worker mid-analysis costs nothing to reschedule, because of how the JRay queue works: an item only leaves /Tasks/Pending once its results are pushed (push_truth). A worker stopped mid-scene_analyze simply leaves that item Pending — next lock picks it up again. No re-queue bookkeeping.

Two small correctness requirements (the only worker changes needed):

1. Never push a partial result. Already true — push_truth runs only after scene_analyze returns; a killed run pushes nothing. ✓ (keep it that way).
2. Clean up on signal. process_item writes a temp filtered-gallery file and unlinks it in a finally; a SIGKILL skips finally. Fix: write temps under a dir the worker wipes on start, and/or a SIGTERM handler that unlinks before exit. Minor.

Accepted trade-off: a partially-analysed title restarts from scratch next lock. Fine for an overnight/idle workload; no mid-video checkpointing.

The end-to-end UX

# once: build + install for your GPU + model
./scripts/build_install.py install.yaml
    # detects Fedora/Arch, ensures deps, compiles, installs units under ~/.local

# once: set your keys, enable the gate
$EDITOR ~/.local/share/sae-worker/.env     # JELLYFIN_API_KEY, TMDB_API_KEY
systemctl --user enable --now sae-lock-gate.service

# from then on: nothing. Lock your screen → it analyses. Unlock → it stops.

No Docker, no GPU passthrough config, no media re-mounting — the worker sees the same filesystem and GPU as everything else on the box.

Implementation plan (follow-up commits)

Ordered so each step stands alone:

1. installer skeleton — scripts/build_install.py: parse install.yaml, distro detect, dependency check/print (start with cpu platform so it builds without a GPU), cmake+build, copy into prefix.
2. supervisor + cleanup — scripts/service.py (worker loop + gallery-scan timer + SIGTERM); temp-file cleanup fix in run_from_jellyfin.py.
3. systemd units + lock gate — generate/install sae-worker.service, sae-lock-gate.service, and the logind lock watcher.
4. gallery/model guard — stamp embedder into gallery.json; startup mismatch check.
5. platform + distro matrix — nvidia/amd backends; dnf/pacman dep lists; ONNX Runtime fetch fallback.
6. docs — README "Run on your idle GPU" section.

Settled decisions

• ONNX Runtime build — the installer fetches the ROCm ORT release. It serves the amd platform, and its CPU execution provider covers the cpu smoke-test fallback too, so one download handles both. (nvidia uses raw TRT and doesn't need ORT.)
• dnf/pacman invocation — auto-install. The installer runs sudo dnf install … / sudo pacman -S … itself (prompting for sudo), rather than only printing the command. It still prints what it's about to install first.
• Distro coverage — Fedora + Arch only for now. Debian/Ubuntu (apt) is out of scope.

Open questions

(none blocking — the spec above is buildable as-is.)