From 7312aceded80a1107b952d5681b2fd34f14ddbd3 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Sun, 30 Aug 2026 10:48:54 +0200 Subject: [PATCH] Get the scene model onto the devices that need it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The decoder can load from a path; nothing yet put a file at one. Four packaging routes, and one lookup that finds the result. ## Not `include_bytes!`, unlike the instance model The instance model is 11 MB and compiled in, which was the right call for it: Android hands the app no filesystem path (ARCH §6.9) and 11 MB is tolerable. The scene model is 24 MB, and 35 MB of constants in the binary is paid by every install whether or not the tab is ever opened. So it follows `models/face/` instead — carried as an APK asset, unpacked once at first launch into the shared directory a desktop install already uses, after which every lookup finds it where it finds a desktop user's. Assets are stored rather than deflated in the APK, so unpacking is a copy rather than an inflate. `embedded-scene-model` exists for the desktop build with nowhere else to read from, and for tests wanting the real graph. Off by default, which is the asymmetry with `embedded-model` and the reason for a separate feature. ## Three files, all or none `scene_model` insists on the graph, its vocabulary and the category descriptor together, for the reason `face_models` insists on its pair: a graph alone decodes to 150 anonymous channels. Reporting the set missing beats starting and failing at the first inference. ## The two model sets are not the same kind of thing `install_bundled_models` now carries both, and the distinction is worth keeping in view. Face weights are absent from the repository *by design* — the InsightFace grant is research-only (docs/faces.md §2) — so a build carrying none is ordinary. The scene model is committed, so a build carrying none means a checkout without `git lfs pull`. Neither is fatal. A photo editor that refuses to start over a missing grading feature is worse than one that starts without it, so both report themselves unavailable exactly as face indexing already did. The LFS-pointer guards apply to the `.onnx` only. The vocabulary and the descriptor are legitimately a few kilobytes, and a size check that fails on them would be a guard against the wrong thing. `scene_model` is exported ahead of the tab that will consume it so the packaging added here has something to be verified against — assets written where no lookup looks would be a silent mistake for as long as the tab took to arrive. Co-Authored-By: Claude Opus 5 (1M context) --- apps/darkroom-android/src/lib.rs | 79 +++++++++++++------ docker/android/assemble-apk.sh | 44 ++++++++--- packaging/PKGBUILD | 19 +++++ packaging/flatpak/paris.tourolle.darkroom.yml | 15 ++++ ui/dr-ui/src/lib.rs | 9 +++ ui/dr-ui/src/library.rs | 31 ++++++++ 6 files changed, 160 insertions(+), 37 deletions(-) diff --git a/apps/darkroom-android/src/lib.rs b/apps/darkroom-android/src/lib.rs index f8c417f..ac0b0ec 100644 --- a/apps/darkroom-android/src/lib.rs +++ b/apps/darkroom-android/src/lib.rs @@ -60,7 +60,7 @@ fn android_main(app: slint::android::AndroidApp) { } // After the data dir and before anything asks whether a model is present. - install_bundled_face_models(&app); + install_bundled_models(&app); // Before `init_with_event_listener`, which takes `app` by value and is the // last moment anything can ask the activity a question. Not an ordering @@ -126,38 +126,65 @@ fn android_main(app: slint::android::AndroidApp) { } } -/// Unpack the face models the APK carries, if it carries any. +/// Unpack the models the APK carries, if it carries any. /// /// # Why Android needs this and no other platform does /// -/// The weights are not a build input and are not in the repository — the -/// InsightFace grant is research-only and incompatible with this project's -/// licence (docs/faces.md §2), so a desktop user fetches them, runs -/// `tools/fix-face-model-shapes.sh` over them, and drops the result into -/// `~/.local/share/darkroom/models/`. **That gesture does not exist on -/// Android.** `internal_data_path` is app-private, `run-as` needs a debuggable -/// build, and there is no picker and no fetch in the app, so a phone had no way -/// to acquire a model at all and face indexing reported itself permanently off. +/// A desktop build reads its models from a path — the account's directory, the +/// shared one, or `$XDG_DATA_DIRS` where a package put them. **Android has no +/// such path.** `internal_data_path` is app-private, `run-as` needs a +/// debuggable build, and an asset inside a package is not a path anything can +/// open (ARCH §6.9), so a phone had no way to reach a model at all. /// -/// So a locally-built APK may carry the pair in `assets/models/`, which -/// `assemble-apk.sh` includes when the tree has them and omits when it does -/// not. Nothing changes about what the repository holds or what a published -/// build could redistribute; this only gives a self-built APK the same route a -/// desktop build has always had. +/// So the APK carries them in `assets/models/` and this copies them out, once, +/// into the same shared directory a desktop install uses. After that every +/// lookup in `dr_ui::library` finds them exactly where it finds a desktop +/// user's. /// -/// Absent assets are the ordinary case, not an error — the same quiet "no model -/// installed" state a fresh desktop install is in. +/// # The two sets are not the same kind of thing +/// +/// **Face weights are absent from the repository by design.** The InsightFace +/// grant is research-only and incompatible with this project's licence +/// (docs/faces.md §2), so a desktop user fetches them, runs +/// `tools/fix-face-model-shapes.sh` over them, and drops the result in. A build +/// that carries none is the ordinary case and face indexing simply stays off. +/// +/// **The scene model is committed** (AGPL, compatible — `models/LICENCE.md`), +/// so a build carrying none means a checkout without `git lfs pull` rather than +/// a deliberate omission. It is still not an error here: the scene tab reports +/// itself unavailable the same way face indexing does, because a photo editor +/// that refuses to start over a missing grading feature is worse than one that +/// starts without it. +/// +/// # Why it is not `include_bytes!` like the instance model +/// +/// Size. The instance model is 11 MB and compiled in; the scene model is 24 MB +/// on top of that, and a 35 MB constant in the binary is paid by every install +/// whether or not the tab is opened. Assets are also *stored* rather than +/// deflated in the APK (see `assemble-apk.sh`), so unpacking is a copy rather +/// than an inflate. #[cfg(target_os = "android")] -fn install_bundled_face_models(app: &slint::android::AndroidApp) { +fn install_bundled_models(app: &slint::android::AndroidApp) { use std::io::Read; - // The **shape-fixed** names, matching what `library::face_models` looks - // for: tract cannot parse either InsightFace graph with its dynamic input - // dimension, so what ships here has already been through - // `tools/fix-face-model-shapes.sh`. - const BUNDLED: [(&std::ffi::CStr, &str); 2] = [ + // The face names are the **shape-fixed** exports, matching what + // `library::face_models` looks for: tract cannot parse either InsightFace + // graph with its dynamic input dimension, so what ships here has already + // been through `tools/fix-face-model-shapes.sh`. + // + // The scene entries are three files rather than one because the graph alone + // decodes to 150 anonymous channels — `library::scene_model` wants the + // vocabulary and the category descriptor beside it, and requires all three + // before it reports the tab available. + const BUNDLED: [(&std::ffi::CStr, &str); 5] = [ (c"models/scrfd_500m_640.onnx", "scrfd_500m_640.onnx"), (c"models/arcface_mbf_b1.onnx", "arcface_mbf_b1.onnx"), + (c"models/yolo26s-sem-ade20k.onnx", "yolo26s-sem-ade20k.onnx"), + ( + c"models/yolo26s-sem-ade20k.classes.json", + "yolo26s-sem-ade20k.classes.json", + ), + (c"models/categories.txt", "categories.txt"), ]; let dir = dr_ui::shared_face_models_dir(); @@ -172,7 +199,7 @@ fn install_bundled_face_models(app: &slint::android::AndroidApp) { continue; } let Some(mut asset) = assets.open(asset_path) else { - log::info!("no bundled {name} in this APK; face indexing stays off"); + log::info!("no bundled {name} in this APK; the feature needing it stays off"); continue; }; let mut bytes = Vec::new(); @@ -185,8 +212,8 @@ fn install_bundled_face_models(app: &slint::android::AndroidApp) { return; } // Written under a temporary name and renamed, because - // `library::face_models` decides face indexing is available on - // `is_file()` alone. A truncated write — the process backgrounded and + // `library::face_models` and `library::scene_model` both decide a + // feature is available on `is_file()` alone. A truncated write — the process backgrounded and // killed mid-copy — would otherwise leave a file that passes that test // and fails inside tract, reported to the user as a broken model rather // than a missing one. diff --git a/docker/android/assemble-apk.sh b/docker/android/assemble-apk.sh index 6e11e09..351facd 100755 --- a/docker/android/assemble-apk.sh +++ b/docker/android/assemble-apk.sh @@ -255,25 +255,37 @@ fi cp "${SO}" "${OUT}/staging/lib/${ABI}/libdarkroom.so" cp "${DEX}" "${OUT}/staging/classes.dex" -# The face models. Android has no other route to one — app-private storage is -# not user-reachable and the in-app fetch is unbuilt (docs/faces.md §2.2a) — so +# The models. Android has no other route to one — app-private storage is not +# user-reachable and the in-app fetch is unbuilt (docs/faces.md §2.2a) — so # they go in the APK and `android_main` unpacks them on first launch. The -# source is `models/face/`, shared with the Arch package rather than living -# under this one platform's directory. +# sources are `models/face/` and `models/scene/`, shared with the Arch package +# rather than living under this one platform's directory. +# +# Two directories, and they are not the same kind of thing. The face weights +# are absent from most checkouts by design (research-only grant), so finding +# none is ordinary. The scene model is committed, so finding none means a +# broken checkout — but this script still only warns, because the failure it +# would otherwise cause is at APK build time on a machine that may legitimately +# be building the face-less variant. # # Through the staging directory rather than aapt2's `-A`: the .so and the dex # already go in with `zip` below, and one mechanism for "extra files in the # APK" is easier to follow than two. -ASSETS="${REPO}/models/face" +# # Cleared first: a previous run that died between staging and cleanup would # otherwise leave models in the APK that are no longer in the tree. rm -rf "${OUT}/staging/assets" -if compgen -G "${ASSETS}/*.onnx" >/dev/null; then +mkdir -p "${OUT}/staging/assets/models" +_bundled="" +for _dir in face scene; do + ASSETS="${REPO}/models/${_dir}" + compgen -G "${ASSETS}/*.onnx" >/dev/null || continue # An LFS pointer is ~130 bytes and looks exactly like a model to `cp`. Left # unchecked it reaches the device and fails inside tract, which reports a # broken graph rather than a clone that needs `git lfs pull`. Same guard # dr-segment's build script applies to yolo26n-seg.onnx, and the same - # reason. + # reason. Only the weights are checked: the vocabulary and the category + # descriptor beside them are legitimately a few kilobytes. for m in "${ASSETS}"/*.onnx; do if [[ "$(stat -c%s "${m}")" -lt 100000 ]]; then echo "error: $(basename "${m}") is $(stat -c%s "${m}") bytes — an LFS pointer, not a model." >&2 @@ -281,11 +293,21 @@ if compgen -G "${ASSETS}/*.onnx" >/dev/null; then exit 1 fi done - mkdir -p "${OUT}/staging/assets/models" - cp "${ASSETS}"/*.onnx "${OUT}/staging/assets/models/" - echo " assets: $(ls "${ASSETS}" | grep '\.onnx$' | tr '\n' ' ')" + # The scene model is three files: the graph, its vocabulary, and the + # category descriptor. All three are needed to decode anything, so they + # travel together; README.md is documentation and stays out of the APK. + for f in "${ASSETS}"/*; do + case "$(basename "${f}")" in + README.md) continue ;; + esac + cp "${f}" "${OUT}/staging/assets/models/" + _bundled="${_bundled} $(basename "${f}")" + done +done +if [[ -n "${_bundled}" ]]; then + echo " assets:${_bundled}" else - echo " assets: no face models found (face indexing will be off on the device)" + echo " assets: no models found (face indexing and the scene tab will be off on the device)" fi # -0 "" stores the .so without compression so Android can mmap it directly diff --git a/packaging/PKGBUILD b/packaging/PKGBUILD index f5fde21..5b051a6 100644 --- a/packaging/PKGBUILD +++ b/packaging/PKGBUILD @@ -69,4 +69,23 @@ package() { fi install -Dm644 "${_src}" "${pkgdir}/usr/share/darkroom/models/${_m}" done + + # The scene model, for the per-category grades. Unlike the face weights + # this one is in the repository — AGPL, and GPLv3 §13 permits the + # combination (models/LICENCE.md) — so it is installed unconditionally and + # a pointer here is a broken checkout rather than a licence decision. + # + # Three files: the graph, its vocabulary, and the category descriptor + # grouping ADE20K's 150 classes into what the tab shows. All three, because + # dr_ui::library::scene_model reports the tab unavailable without any one + # of them. Only the graph gets the pointer check — the other two are + # legitimately a few kilobytes. + _src="models/scene/yolo26s-sem-ade20k.onnx" + if [[ "$(stat -c%s "${_src}")" -lt 100000 ]]; then + echo "error: the scene model is an LFS pointer — run: git lfs pull" >&2 + return 1 + fi + for _m in yolo26s-sem-ade20k.onnx yolo26s-sem-ade20k.classes.json categories.txt; do + install -Dm644 "models/scene/${_m}" "${pkgdir}/usr/share/darkroom/models/${_m}" + done } diff --git a/packaging/flatpak/paris.tourolle.darkroom.yml b/packaging/flatpak/paris.tourolle.darkroom.yml index c6b9c27..23dec17 100644 --- a/packaging/flatpak/paris.tourolle.darkroom.yml +++ b/packaging/flatpak/paris.tourolle.darkroom.yml @@ -165,6 +165,21 @@ modules: install -Dm644 "models/face/$m" "/app/share/darkroom/models/$m" done + # The scene model, for the per-category grades. In the repository, unlike + # the face weights — AGPL, and GPLv3 §13 permits the combination + # (models/LICENCE.md) — so it installs unconditionally. Three files: the + # graph, its vocabulary, and the category descriptor; dr_ui reports the + # tab unavailable without any one of them. The pointer check is on the + # graph alone, the other two being legitimately small. + - | + if [ "$(stat -c%s models/scene/yolo26s-sem-ade20k.onnx)" -lt 100000 ]; then + echo "error: the scene model is an LFS pointer — run: git lfs pull" >&2 + exit 1 + fi + for m in yolo26s-sem-ade20k.onnx yolo26s-sem-ade20k.classes.json categories.txt; do + install -Dm644 "models/scene/$m" "/app/share/darkroom/models/$m" + done + - install -Dm644 README.md /app/share/doc/darkroom/README.md sources: diff --git a/ui/dr-ui/src/lib.rs b/ui/dr-ui/src/lib.rs index e7c3364..70d70a2 100644 --- a/ui/dr-ui/src/lib.rs +++ b/ui/dr-ui/src/lib.rs @@ -73,6 +73,15 @@ pub use develop::DevelopSession; /// opened — see `library::shared_face_models_dir`. pub use library::shared_face_models_dir; +/// The scene model's three files, wherever this device keeps them. +/// +/// Public for the same reason as the directory above: the pieces that reach for +/// a model are not all inside this crate. Exported ahead of the scene tab that +/// will consume it so that the packaging and unpacking added alongside it have +/// something to be verified against — `assemble-apk.sh` writing files no lookup +/// looks for would be a silent mistake for as long as the tab took to arrive. +pub use library::scene_model; + pub mod launch; pub mod launch_ui; diff --git a/ui/dr-ui/src/library.rs b/ui/dr-ui/src/library.rs index c14796f..ab4562c 100644 --- a/ui/dr-ui/src/library.rs +++ b/ui/dr-ui/src/library.rs @@ -3846,6 +3846,37 @@ pub fn face_models(account: &Account) -> Option<(PathBuf, PathBuf)> { .or_else(|| system_face_models_dirs().into_iter().find_map(pair)) } +/// The scene model, its vocabulary and its category descriptor, if all three +/// are present. +/// +/// All three or none, for the same reason `face_models` insists on its pair: +/// the graph alone decodes to 150 anonymous channels, and a descriptor naming +/// classes a different model does not have is refused by +/// `dr_segment::scene::parse_categories` anyway. Reporting the set missing is +/// more useful than starting and failing at the first inference. +/// +/// Searched in the same three places, most specific first — the account's own +/// directory, the shared one, then wherever a package installed them. Android +/// only ever finds the second, which is where `install_bundled_models` unpacks +/// the APK's copy before any store opens. +/// +/// Unlike the face weights this model *is* in the repository, so a desktop +/// build from a complete checkout has it. Absent means either a checkout +/// without `git lfs pull` or a package that chose not to carry 24 MB, and the +/// scene tab reports itself unavailable rather than the app refusing to run. +pub fn scene_model(account: &Account) -> Option<(PathBuf, PathBuf, PathBuf)> { + fn set(dir: PathBuf) -> Option<(PathBuf, PathBuf, PathBuf)> { + let model = dir.join("yolo26s-sem-ade20k.onnx"); + let classes = dir.join("yolo26s-sem-ade20k.classes.json"); + let categories = dir.join("categories.txt"); + (model.is_file() && classes.is_file() && categories.is_file()) + .then_some((model, classes, categories)) + } + set(face_models_dir(account)) + .or_else(|| set(shared_face_models_dir())) + .or_else(|| system_face_models_dirs().into_iter().find_map(set)) +} + /// Where a *package* may have installed the models. /// /// `$XDG_DATA_DIRS` rather than a hard-coded `/usr/share`, because that is the