dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Get the scene model onto the devices that need it

Browse Source 
The decoder can load from a path; nothing yet put a file at one. Four
packaging routes, and one lookup that finds the result.

## Not `include_bytes!`, unlike the instance model

The instance model is 11 MB and compiled in, which was the right call for
it: Android hands the app no filesystem path (ARCH §6.9) and 11 MB is
tolerable. The scene model is 24 MB, and 35 MB of constants in the binary
is paid by every install whether or not the tab is ever opened.

So it follows `models/face/` instead — carried as an APK asset, unpacked
once at first launch into the shared directory a desktop install already
uses, after which every lookup finds it where it finds a desktop user's.
Assets are stored rather than deflated in the APK, so unpacking is a copy
rather than an inflate.

`embedded-scene-model` exists for the desktop build with nowhere else to
read from, and for tests wanting the real graph. Off by default, which is
the asymmetry with `embedded-model` and the reason for a separate
feature.

## Three files, all or none

`scene_model` insists on the graph, its vocabulary and the category
descriptor together, for the reason `face_models` insists on its pair: a
graph alone decodes to 150 anonymous channels. Reporting the set missing
beats starting and failing at the first inference.

## The two model sets are not the same kind of thing

`install_bundled_models` now carries both, and the distinction is worth
keeping in view. Face weights are absent from the repository *by design*
— the InsightFace grant is research-only (docs/faces.md §2) — so a build
carrying none is ordinary. The scene model is committed, so a build
carrying none means a checkout without `git lfs pull`.

Neither is fatal. A photo editor that refuses to start over a missing
grading feature is worse than one that starts without it, so both report
themselves unavailable exactly as face indexing already did.

The LFS-pointer guards apply to the `.onnx` only. The vocabulary and the
descriptor are legitimately a few kilobytes, and a size check that fails
on them would be a guard against the wrong thing.

`scene_model` is exported ahead of the tab that will consume it so the
packaging added here has something to be verified against — assets
written where no lookup looks would be a silent mistake for as long as
the tab took to arrive.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dtourolle
2026-08-30 10:48:54 +02:00
co-authored by Claude Opus 5
parent 8df6000e4b
commit 7312aceded
6 changed files with 160 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/darkroom-android/src/lib.rs
+53 -26
View File
@@ -60,7 +60,7 @@ fn android_main(app: slint::android::AndroidApp) {
}
// After the data dir and before anything asks whether a model is present.
install_bundled_face_models(&app);
install_bundled_models(&app);
// Before `init_with_event_listener`, which takes `app` by value and is the
// last moment anything can ask the activity a question. Not an ordering
@@ -126,38 +126,65 @@ fn android_main(app: slint::android::AndroidApp) {
}
}
/// Unpack the face models the APK carries, if it carries any.
/// Unpack the models the APK carries, if it carries any.
///
/// # Why Android needs this and no other platform does
///
/// The weights are not a build input and are not in the repository — the
/// InsightFace grant is research-only and incompatible with this project's
/// licence (docs/faces.md §2), so a desktop user fetches them, runs
/// `tools/fix-face-model-shapes.sh` over them, and drops the result into
/// `~/.local/share/darkroom/models/`. **That gesture does not exist on
/// Android.** `internal_data_path` is app-private, `run-as` needs a debuggable
/// build, and there is no picker and no fetch in the app, so a phone had no way
/// to acquire a model at all and face indexing reported itself permanently off.
/// A desktop build reads its models from a path — the account's directory, the
/// shared one, or `$XDG_DATA_DIRS` where a package put them. **Android has no
/// such path.** `internal_data_path` is app-private, `run-as` needs a
/// debuggable build, and an asset inside a package is not a path anything can
/// open (ARCH §6.9), so a phone had no way to reach a model at all.
///
/// So a locally-built APK may carry the pair in `assets/models/`, which
/// `assemble-apk.sh` includes when the tree has them and omits when it does
/// not. Nothing changes about what the repository holds or what a published
/// build could redistribute; this only gives a self-built APK the same route a
/// desktop build has always had.
/// So the APK carries them in `assets/models/` and this copies them out, once,
/// into the same shared directory a desktop install uses. After that every
/// lookup in `dr_ui::library` finds them exactly where it finds a desktop
/// user's.
///
/// Absent assets are the ordinary case, not an error — the same quiet "no model
/// installed" state a fresh desktop install is in.
/// # The two sets are not the same kind of thing
///
/// **Face weights are absent from the repository by design.** The InsightFace
/// grant is research-only and incompatible with this project's licence
/// (docs/faces.md §2), so a desktop user fetches them, runs
/// `tools/fix-face-model-shapes.sh` over them, and drops the result in. A build
/// that carries none is the ordinary case and face indexing simply stays off.
///
/// **The scene model is committed** (AGPL, compatible — `models/LICENCE.md`),
/// so a build carrying none means a checkout without `git lfs pull` rather than
/// a deliberate omission. It is still not an error here: the scene tab reports
/// itself unavailable the same way face indexing does, because a photo editor
/// that refuses to start over a missing grading feature is worse than one that
/// starts without it.
///
/// # Why it is not `include_bytes!` like the instance model
///
/// Size. The instance model is 11 MB and compiled in; the scene model is 24 MB
/// on top of that, and a 35 MB constant in the binary is paid by every install
/// whether or not the tab is opened. Assets are also *stored* rather than
/// deflated in the APK (see `assemble-apk.sh`), so unpacking is a copy rather
/// than an inflate.
#[cfg(target_os = "android")]
fn install_bundled_face_models(app: &slint::android::AndroidApp) {
fn install_bundled_models(app: &slint::android::AndroidApp) {
use std::io::Read;
// The **shape-fixed** names, matching what `library::face_models` looks
// for: tract cannot parse either InsightFace graph with its dynamic input
// dimension, so what ships here has already been through
// `tools/fix-face-model-shapes.sh`.
const BUNDLED: [(&std::ffi::CStr, &str); 2] = [
// The face names are the **shape-fixed** exports, matching what
// `library::face_models` looks for: tract cannot parse either InsightFace
// graph with its dynamic input dimension, so what ships here has already
// been through `tools/fix-face-model-shapes.sh`.
//
// The scene entries are three files rather than one because the graph alone
// decodes to 150 anonymous channels — `library::scene_model` wants the
// vocabulary and the category descriptor beside it, and requires all three
// before it reports the tab available.
const BUNDLED: [(&std::ffi::CStr, &str); 5] = [
(c"models/scrfd_500m_640.onnx", "scrfd_500m_640.onnx"),
(c"models/arcface_mbf_b1.onnx", "arcface_mbf_b1.onnx"),
(c"models/yolo26s-sem-ade20k.onnx", "yolo26s-sem-ade20k.onnx"),
(
c"models/yolo26s-sem-ade20k.classes.json",
"yolo26s-sem-ade20k.classes.json",
),
(c"models/categories.txt", "categories.txt"),
];
let dir = dr_ui::shared_face_models_dir();
@@ -172,7 +199,7 @@ fn install_bundled_face_models(app: &slint::android::AndroidApp) {
continue;
}
let Some(mut asset) = assets.open(asset_path) else {
log::info!("no bundled {name} in this APK; face indexing stays off");
log::info!("no bundled {name} in this APK; the feature needing it stays off");
continue;
};
let mut bytes = Vec::new();
@@ -185,8 +212,8 @@ fn install_bundled_face_models(app: &slint::android::AndroidApp) {
return;
}
// Written under a temporary name and renamed, because
// `library::face_models` decides face indexing is available on
// `is_file()` alone. A truncated write — the process backgrounded and
// `library::face_models` and `library::scene_model` both decide a
// feature is available on `is_file()` alone. A truncated write — the process backgrounded and
// killed mid-copy — would otherwise leave a file that passes that test
// and fails inside tract, reported to the user as a broken model rather
// than a missing one.
docker/android/assemble-apk.sh
+33 -11
View File
@@ -255,25 +255,37 @@ fi
cp "${SO}" "${OUT}/staging/lib/${ABI}/libdarkroom.so"
cp "${DEX}" "${OUT}/staging/classes.dex"
# The face models. Android has no other route to one — app-private storage is
# not user-reachable and the in-app fetch is unbuilt (docs/faces.md §2.2a) — so
# The models. Android has no other route to one — app-private storage is not
# user-reachable and the in-app fetch is unbuilt (docs/faces.md §2.2a) — so
# they go in the APK and `android_main` unpacks them on first launch. The
# source is `models/face/`, shared with the Arch package rather than living
# under this one platform's directory.
# sources are `models/face/` and `models/scene/`, shared with the Arch package
# rather than living under this one platform's directory.
#
# Two directories, and they are not the same kind of thing. The face weights
# are absent from most checkouts by design (research-only grant), so finding
# none is ordinary. The scene model is committed, so finding none means a
# broken checkout — but this script still only warns, because the failure it
# would otherwise cause is at APK build time on a machine that may legitimately
# be building the face-less variant.
#
# Through the staging directory rather than aapt2's `-A`: the .so and the dex
# already go in with `zip` below, and one mechanism for "extra files in the
# APK" is easier to follow than two.
ASSETS="${REPO}/models/face"
#
# Cleared first: a previous run that died between staging and cleanup would
# otherwise leave models in the APK that are no longer in the tree.
rm -rf "${OUT}/staging/assets"
if compgen -G "${ASSETS}/*.onnx" >/dev/null; then
mkdir -p "${OUT}/staging/assets/models"
_bundled=""
for _dir in face scene; do
ASSETS="${REPO}/models/${_dir}"
compgen -G "${ASSETS}/*.onnx" >/dev/null || continue
# An LFS pointer is ~130 bytes and looks exactly like a model to `cp`. Left
# unchecked it reaches the device and fails inside tract, which reports a
# broken graph rather than a clone that needs `git lfs pull`. Same guard
# dr-segment's build script applies to yolo26n-seg.onnx, and the same
# reason.
# reason. Only the weights are checked: the vocabulary and the category
# descriptor beside them are legitimately a few kilobytes.
for m in "${ASSETS}"/*.onnx; do
if [[ "$(stat -c%s "${m}")" -lt 100000 ]]; then
echo "error: $(basename "${m}") is $(stat -c%s "${m}") bytes — an LFS pointer, not a model." >&2
@@ -281,11 +293,21 @@ if compgen -G "${ASSETS}/*.onnx" >/dev/null; then
exit 1
fi
done
mkdir -p "${OUT}/staging/assets/models"
cp "${ASSETS}"/*.onnx "${OUT}/staging/assets/models/"
echo " assets: $(ls "${ASSETS}" | grep '\.onnx$' | tr '\n' ' ')"
# The scene model is three files: the graph, its vocabulary, and the
# category descriptor. All three are needed to decode anything, so they
# travel together; README.md is documentation and stays out of the APK.
for f in "${ASSETS}"/*; do
case "$(basename "${f}")" in
README.md) continue ;;
esac
cp "${f}" "${OUT}/staging/assets/models/"
_bundled="${_bundled} $(basename "${f}")"
done
done
if [[ -n "${_bundled}" ]]; then
echo " assets:${_bundled}"
else
echo " assets: no face models found (face indexing will be off on the device)"
echo " assets: no models found (face indexing and the scene tab will be off on the device)"
fi
# -0 "" stores the .so without compression so Android can mmap it directly
packaging/PKGBUILD
+19
View File
@@ -69,4 +69,23 @@ package() {
fi
install -Dm644 "${_src}" "${pkgdir}/usr/share/darkroom/models/${_m}"
done
# The scene model, for the per-category grades. Unlike the face weights
# this one is in the repository — AGPL, and GPLv3 §13 permits the
# combination (models/LICENCE.md) — so it is installed unconditionally and
# a pointer here is a broken checkout rather than a licence decision.
#
# Three files: the graph, its vocabulary, and the category descriptor
# grouping ADE20K's 150 classes into what the tab shows. All three, because
# dr_ui::library::scene_model reports the tab unavailable without any one
# of them. Only the graph gets the pointer check — the other two are
# legitimately a few kilobytes.
_src="models/scene/yolo26s-sem-ade20k.onnx"
if [[ "$(stat -c%s "${_src}")" -lt 100000 ]]; then
echo "error: the scene model is an LFS pointer — run: git lfs pull" >&2
return 1
fi
for _m in yolo26s-sem-ade20k.onnx yolo26s-sem-ade20k.classes.json categories.txt; do
install -Dm644 "models/scene/${_m}" "${pkgdir}/usr/share/darkroom/models/${_m}"
done
}
packaging/flatpak/paris.tourolle.darkroom.yml
+15
View File
@@ -165,6 +165,21 @@ modules:
install -Dm644 "models/face/$m" "/app/share/darkroom/models/$m"
done
# The scene model, for the per-category grades. In the repository, unlike
# the face weights — AGPL, and GPLv3 §13 permits the combination
# (models/LICENCE.md) — so it installs unconditionally. Three files: the
# graph, its vocabulary, and the category descriptor; dr_ui reports the
# tab unavailable without any one of them. The pointer check is on the
# graph alone, the other two being legitimately small.
- |
if [ "$(stat -c%s models/scene/yolo26s-sem-ade20k.onnx)" -lt 100000 ]; then
echo "error: the scene model is an LFS pointer — run: git lfs pull" >&2
exit 1
fi
for m in yolo26s-sem-ade20k.onnx yolo26s-sem-ade20k.classes.json categories.txt; do
install -Dm644 "models/scene/$m" "/app/share/darkroom/models/$m"
done
- install -Dm644 README.md /app/share/doc/darkroom/README.md
sources:
ui/dr-ui/src/lib.rs
+9
View File
@@ -73,6 +73,15 @@ pub use develop::DevelopSession;
/// opened — see `library::shared_face_models_dir`.
pub use library::shared_face_models_dir;
/// The scene model's three files, wherever this device keeps them.
///
/// Public for the same reason as the directory above: the pieces that reach for
/// a model are not all inside this crate. Exported ahead of the scene tab that
/// will consume it so that the packaging and unpacking added alongside it have
/// something to be verified against — `assemble-apk.sh` writing files no lookup
/// looks for would be a silent mistake for as long as the tab took to arrive.
pub use library::scene_model;
pub mod launch;
pub mod launch_ui;
ui/dr-ui/src/library.rs
+31
View File
@@ -3846,6 +3846,37 @@ pub fn face_models(account: &Account) -> Option<(PathBuf, PathBuf)> {
.or_else(|| system_face_models_dirs().into_iter().find_map(pair))
}
/// The scene model, its vocabulary and its category descriptor, if all three
/// are present.
///
/// All three or none, for the same reason `face_models` insists on its pair:
/// the graph alone decodes to 150 anonymous channels, and a descriptor naming
/// classes a different model does not have is refused by
/// `dr_segment::scene::parse_categories` anyway. Reporting the set missing is
/// more useful than starting and failing at the first inference.
///
/// Searched in the same three places, most specific first — the account's own
/// directory, the shared one, then wherever a package installed them. Android
/// only ever finds the second, which is where `install_bundled_models` unpacks
/// the APK's copy before any store opens.
///
/// Unlike the face weights this model *is* in the repository, so a desktop
/// build from a complete checkout has it. Absent means either a checkout
/// without `git lfs pull` or a package that chose not to carry 24 MB, and the
/// scene tab reports itself unavailable rather than the app refusing to run.
pub fn scene_model(account: &Account) -> Option<(PathBuf, PathBuf, PathBuf)> {
fn set(dir: PathBuf) -> Option<(PathBuf, PathBuf, PathBuf)> {
let model = dir.join("yolo26s-sem-ade20k.onnx");
let classes = dir.join("yolo26s-sem-ade20k.classes.json");
let categories = dir.join("categories.txt");
(model.is_file() && classes.is_file() && categories.is_file())
.then_some((model, classes, categories))
}
set(face_models_dir(account))
.or_else(|| set(shared_face_models_dir()))
.or_else(|| system_face_models_dirs().into_iter().find_map(set))
}
/// Where a *package* may have installed the models.
///
/// `$XDG_DATA_DIRS` rather than a hard-coded `/usr/share`, because that is the