dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Run each model on the Hexagon in the form measured to hold it

Browse Source 
The engine knew f32 and int8, and gave the Hexagon int8 for every role it
served. Measured on the tablet itself (inference.md §1.5), int8 lost
5% of the detector's faces at 40-80 px, moved the landmarks 1.5 px,
emptied the segmenter's scores and cost the denoiser 5-9 dB; fp16 the HTP
refuses outright. `Form` gains A16W8 and A16W16, and `Rung::form` now
names one per role: detectors and landmarks A16W8, the segmenter, scene
model, border filler and denoiser A16W16, XFeat int8. The embedder and
the eye classifiers stay on the CPU.

Each loader resolves its `<stem>.<form>.onnx` sibling; the segmenter and
XFeat, compiled into the binary, embed their quantised forms on Android
only and pick through `choose_embedded`. The probe, the compile step and
the cache fingerprint follow the form instead of assuming int8. Detectors
on the new form write `scrfd_*_a16+w600k_mbf`, and `model_ids` answers
for all three spellings.

On the tablet (ORT 1.29 + QNN 2.42), each shipped file against f32 on the
same inputs, and against the CPU's f32 time:
  SCRFD 500m/2.5g/10g  A16W8   100% of faces in every band   4.2/5.1/9.0 ms vs 17/56/198
  landmarks            A16W8   0.25 px in the 192 crop        0.5 ms vs 2.8
  YOLO26n-seg          A16W16  98.2% found, mask IoU 0.994    12.9 ms vs 90
  scene model          A16W16  98.9% of cells agree           15 ms vs 151
  MI-GAN               A16W16  41 dB from f32 in the fill     87 ms vs 488
  XFeat                int8    pano alignment 0.45 px (f32's own spread 0.41)  6.5 ms vs 58
  denoiser             A16W16  0.00 dB at every ISO            95 ms vs 1510 a tile
Face numbers are over public COCO val2017 photographs, not a library.

The APK carries the siblings (BUNDLED 15 -> 19; the old int8 detectors
removed), about 43 MB more. The Windows installer and its CI count skip
them; the Arch and Flatpak packages list their files and never had them.
The ladder example takes a role per model, which is how the per-role
forms above were seen landing on the NPU from the real probe.
This commit is contained in:
dtourolle
2026-10-04 03:45:46 -04:00
parent 0e6ac09fd5
commit 5a8c3e4c40
39 changed files with 550 additions and 208 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/dr-inference-engine/examples/ladder.rs
+39 -19
View File
@@ -4,12 +4,14 @@
//!
//! DARKROOM_ORT_DIR=/usr/lib \
//! cargo run --release -p dr-inference-engine --features native,tract \
//! --example ladder -- CACHE_DIR models/face/scrfd_500m_640.onnx [MODEL.onnx ...]
//! --example ladder -- CACHE_DIR models/face/scrfd_500m_640.onnx [ROLE=MODEL.onnx ...]
//!
//! Every model named is a `Detector` for the config's purposes, which is
//! enough to see the rung taken, the engines compiled and a session land
//! on it. Delete `CACHE_DIR` to see the first run again; keep it to see the
//! second.
//! A bare path is a `Detector`; `denoiser=…`, `scene=…`, `inpainter=…`,
//! `landmarks=…` (any `Role`, lower case) says otherwise, so a device can
//! show each role taking its own form (inference.md §1.5). Each is opened
//! through `resolve_model`, as the app opens it, and the line says which
//! form and which rung it landed on. Delete `CACHE_DIR` to see the first
//! run again; keep it to see the second.
use std::path::PathBuf;
use std::time::{Duration, Instant};
@@ -17,7 +19,10 @@ use std::time::{Duration, Instant};
fn main() {
env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();
let mut args = std::env::args_os().skip(1).map(PathBuf::from);
let (Some(cache_dir), models) = (args.next(), args.collect::<Vec<_>>()) else {
let (Some(cache_dir), models) = (
args.next(),
args.map(|a| role_and_path(&a)).collect::<Vec<_>>(),
) else {
eprintln!("usage: ladder CACHE_DIR MODEL.onnx [MODEL.onnx ...]");
std::process::exit(2);
};
@@ -34,10 +39,7 @@ fn main() {
dr_inference_engine::init(dr_inference_engine::Config {
runtime_dirs,
cache_dir: cache_dir.clone(),
models: models
.iter()
.map(|p| (dr_inference_engine::Role::Detector, p.clone()))
.collect(),
models: models.clone(),
embedded: Vec::new(),
ceiling: None,
threads: 0,
@@ -80,21 +82,39 @@ fn main() {
std::thread::sleep(Duration::from_millis(500));
}
for path in &models {
let bytes = std::fs::read(path).expect("read model");
for (role, path) in &models {
let (path, form) = dr_inference_engine::resolve_model(*role, path);
let bytes = std::fs::read(&path).expect("read model");
let t = Instant::now();
let model = dr_inference_engine::open(
dr_inference_engine::Role::Detector,
dr_inference_engine::Form::F32,
&bytes,
)
.expect("open model");
let model = dr_inference_engine::open(*role, form, &bytes).expect("open model");
let acquired = model.acquire().expect("acquire session");
println!(
"{} on {} in {:.2} s",
"{role:?}: {} ({form:?}) on {} in {:.2} s",
path.file_name().unwrap().to_string_lossy(),
acquired.rung().label(),
t.elapsed().as_secs_f64()
);
}
}
/// `denoiser=path` → (Denoiser, path); a bare path is a detector.
fn role_and_path(arg: &std::path::Path) -> (dr_inference_engine::Role, PathBuf) {
use dr_inference_engine::Role::*;
let s = arg.to_string_lossy();
let Some((name, path)) = s.split_once('=') else {
return (Detector, arg.to_path_buf());
};
let role = match name {
"detector" => Detector,
"embedder" => Embedder,
"segmenter" => Segmenter,
"scene" => Scene,
"landmarks" => Landmarks,
"eyes" => EyeClassifier,
"keypoints" => Keypoints,
"inpainter" => Inpainter,
"denoiser" => Denoiser,
other => panic!("no role {other:?}"),
};
(role, PathBuf::from(path))
}
core/dr-inference-engine/src/engines.rs
+5 -5
View File
@@ -8,7 +8,7 @@
use std::path::PathBuf;
use crate::{state, Config, Form, Rung};
use crate::{state, Config, Rung};
enum Source {
File(PathBuf),
@@ -82,10 +82,10 @@ pub fn run() {
(*role, Source::File(path), size)
})
})
.chain(cfg.embedded.iter().filter_map(|(role, bytes)| {
// An embedded model has no int8 sibling to offer a rung that
// wants one; it runs on that rung's fallback.
(rung.serves(*role) && rung.form(*role) == Form::F32).then_some((
.chain(cfg.embedded.iter().filter_map(|(role, form, bytes)| {
// The embedded form the rung wants, if the build carries it;
// a build without it runs that model on the rung's fallback.
(rung.serves(*role) && rung.form(*role) == *form).then_some((
*role,
Source::Bytes(bytes),
bytes.len() as u64,
core/dr-inference-engine/src/lib.rs
+139 -59
View File
@@ -42,25 +42,46 @@ pub enum Role {
/// XFeat, the panorama keypoint detector (docs/dev/panorama.md).
Keypoints,
/// MI-GAN, the panorama border filler (docs/dev/panorama.md §12). Plain
/// convolutions, so any rung serves it; fp16 on TensorRT and int8 on
/// the Hexagon are the point of it.
/// convolutions, so any rung serves it; fp16 on TensorRT and 16-bit
/// activations on the Hexagon (int8 changes the fill, §1.5).
Inpainter,
/// The learned demosaic and denoise on the raw mosaic (docs/dev/denoise.md).
/// fp16 costs it nothing measurable; int8 costs 6–9 dB, because 256
/// levels cannot hold the shadow steps it exists to recover — so the
/// Hexagon does not take it.
/// Hexagon takes it with 16-bit activations and weights (§1.5).
Denoiser,
}
/// Which numeric form of a model a session was built from.
///
/// `Int8` is a different network from `F32` for a detector — it finds a
/// different set of faces — which is why [`form_suffix`] exists and why a
/// The quantised forms are QDQ graphs, per-channel weights, as QNN's HTP
/// takes them (docs/dev/inference.md §1.5): `Int8` is 8-bit activations and
/// weights, `A16W8` 16-bit activations with 8-bit weights, `A16W16` 16-bit
/// both. The Hexagon accepts no float tensor at all, so these are the
/// whole menu; which one a role gets is [`Rung::form`], measured per model.
///
/// A quantised detector is a different network from the f32 one — it finds
/// a different set of faces — which is why [`form_suffix`] exists and why a
/// caller appends it to `model_id`.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub enum Form {
F32,
Int8,
A16W8,
A16W16,
}
impl Form {
/// The infix of the sibling file that holds this form:
/// `scrfd_500m_640.a16w8.onnx` beside `scrfd_500m_640.onnx`.
pub fn file_tag(self) -> Option<&'static str> {
match self {
Form::F32 => None,
Form::Int8 => Some("int8"),
Form::A16W8 => Some("a16w8"),
Form::A16W16 => Some("a16w16"),
}
}
}
/// A rung of the ladder (§2). Ordered: a user override names the highest rung
@@ -81,7 +102,7 @@ pub enum Rung {
/// removed in ONNX Runtime 1.23, so there is no non-compiling AMD rung
/// to fall back to: this one falls back to the CPU.
MiGraphX,
/// Qualcomm's Hexagon NPU through QNN, int8 models only. Android only.
/// Qualcomm's Hexagon NPU through QNN, quantised models only. Android only.
Hexagon,
/// Apple, through CoreML: the Neural Engine, the GPU or the CPU, as
/// CoreML schedules it. macOS only. Compiles an ML Program per model on
@@ -121,23 +142,38 @@ impl Rung {
}
/// The model form this rung wants for a role.
fn form(self, _role: Role) -> Form {
///
/// On the Hexagon, the narrowest form that held each model's accuracy
/// on the tablet itself (§1.5): int8 lost 5% of the detector's faces at
/// 40–80 px, moved the landmarks by 1.5 px and the segmenter's scores
/// to nothing, and the denoiser by 6–9 dB, so those take 16-bit
/// activations; the segmenter, scene model, filler and denoiser also
/// needed 16-bit weights. Only XFeat keeps int8: its panorama alignment
/// moved by no more than f32's own refits do.
pub fn form(self, role: Role) -> Form {
match self {
Rung::Hexagon => Form::Int8,
Rung::Hexagon => match role {
Role::Keypoints => Form::Int8,
Role::Detector | Role::Landmarks => Form::A16W8,
Role::Segmenter | Role::Scene | Role::Inpainter | Role::Denoiser => Form::A16W16,
Role::Embedder | Role::EyeClassifier => Form::F32,
},
_ => Form::F32,
}
}
/// Whether this rung runs `role` at all. The Hexagon takes int8 graphs
/// only, and the embedder is never int8 (§7) — it runs on the CPU
/// beside a detector on the NPU, so its vectors compare across devices.
/// Nor is the denoiser: its int8 form failed the 0.5 dB gate by 6–9 dB
/// (denoise.md §8), so it runs on the CPU there too. CoreML is kept off
/// the embedder for the same reason as the Hexagon: the Neural Engine is
/// fp16, and which unit runs a graph is CoreML's choice.
/// Whether this rung runs `role` at all. The Hexagon takes quantised
/// graphs only, and the embedder is never quantised (§7) — it runs on
/// the CPU beside a detector on the NPU, so its vectors compare across
/// devices; at A16W16 it still missed the 0.999 cosine gate. The eye
/// classifiers stay on the CPU too: a millisecond there, and the two
/// share one role while only one of them held its readings quantised.
/// CoreML is kept off the embedder for the same reason as the Hexagon:
/// the Neural Engine is fp16, and which unit runs a graph is CoreML's
/// choice.
fn serves(self, role: Role) -> bool {
match self {
Rung::Hexagon => !matches!(role, Role::Embedder | Role::Denoiser),
Rung::Hexagon => !matches!(role, Role::Embedder | Role::EyeClassifier),
Rung::CoreMl => role != Role::Embedder,
_ => true,
}
@@ -161,8 +197,10 @@ pub struct Config {
/// The canonical model files on this device, so engines can be compiled
/// ahead of the first request for them.
pub models: Vec<(Role, PathBuf)>,
/// Models compiled into the binary, for the same reason.
pub embedded: Vec<(Role, &'static [u8])>,
/// Models compiled into the binary, for the same reason, each with the
/// form it is. A build that embeds a quantised sibling lists it here
/// beside the f32 graph, and the compile step takes the one the rung wants.
pub embedded: Vec<(Role, Form, &'static [u8])>,
/// The highest rung the user allows; `None` is "the best that works".
pub ceiling: Option<Rung>,
/// ONNX Runtime's intra-op pool; 0 picks from the core count.
@@ -188,10 +226,10 @@ pub struct Status {
}
impl Status {
/// "Hexagon NPU · int8 · ONNX Runtime 1.29" — the settings row's text.
/// "Hexagon NPU · quantised · ONNX Runtime 1.29" — the settings row's text.
pub fn line(&self) -> String {
let form = match self.rung {
Rung::Hexagon => " · int8",
Rung::Hexagon => " · quantised",
Rung::TensorRt | Rung::MiGraphX => " · fp16",
_ => "",
};
@@ -464,26 +502,44 @@ fn current_rung(s: &State) -> Rung {
/// The file to load for `role` under the current selection, and its form.
///
/// A rung that wants int8 gets the `.int8.onnx` sibling of the canonical file
/// if it exists; otherwise the canonical file, on the rung's fallback. A
/// caller adds [`form_suffix`] to the `model_id` it records.
/// A rung that wants a quantised form gets that sibling of the canonical
/// file (`<stem>.a16w8.onnx` and so on, [`Form::file_tag`]) if it exists;
/// otherwise the canonical file, on the rung's fallback. A caller adds
/// [`form_suffix`] to the `model_id` it records.
pub fn resolve_model(role: Role, canonical: &Path) -> (PathBuf, Form) {
let rung = current_rung(&state().lock().unwrap());
if rung.serves(role) && rung.form(role) == Form::Int8 {
let sibling = int8_sibling(canonical);
let want = rung.form(role);
if rung.serves(role) && want != Form::F32 {
let sibling = form_sibling(canonical, want);
if sibling.is_file() {
return (sibling, Form::Int8);
return (sibling, want);
}
}
(canonical.to_path_buf(), Form::F32)
}
fn int8_sibling(canonical: &Path) -> PathBuf {
/// The same choice for a model compiled into the binary: of the forms
/// `offered`, the one the current rung wants for `role`, else the f32 one.
/// `offered` must hold an `F32` entry.
pub fn choose_embedded(role: Role, offered: &[(Form, &'static [u8])]) -> (&'static [u8], Form) {
let rung = current_rung(&state().lock().unwrap());
let want = rung.form(role);
let pick = |form| offered.iter().find(|(f, _)| *f == form);
let (form, bytes) = (rung.serves(role).then(|| pick(want)).flatten())
.or_else(|| pick(Form::F32))
.expect("an embedded model offers its f32 form");
(bytes, *form)
}
fn form_sibling(canonical: &Path, form: Form) -> PathBuf {
let stem = canonical
.file_stem()
.map(|s| s.to_string_lossy().into_owned())
.unwrap_or_default();
canonical.with_file_name(format!("{stem}.int8.onnx"))
match form.file_tag() {
Some(tag) => canonical.with_file_name(format!("{stem}.{tag}.onnx")),
None => canonical.to_path_buf(),
}
}
/// What a form appends to a detector's `model_id` (§7).
@@ -491,6 +547,8 @@ pub fn form_suffix(form: Form) -> &'static str {
match form {
Form::F32 => "",
Form::Int8 => "_i8",
Form::A16W8 => "_a16",
Form::A16W16 => "_a16w16",
}
}
@@ -520,8 +578,8 @@ pub fn open(role: Role, form: Form, bytes: &[u8]) -> Result<Model, Error> {
fn effective_rung(s: &State, selected: Rung, role: Role, form: Form, hash: u64) -> Rung {
let mut rung = selected;
if !rung.serves(role) || rung.form(role) != form {
// The embedder on a Hexagon device, or an f32 detector where the int8
// sibling was missing: neither can go to the NPU.
// The embedder on a Hexagon device, or an f32 detector where the
// quantised sibling was missing: neither can go to the NPU.
rung = rung.fallback();
}
if rung.compiles() && !s.cache.compiled.contains(&engines::key_of(rung, hash)) {
@@ -612,9 +670,10 @@ mod tests {
#[test]
fn the_hexagon_never_takes_the_embedder() {
assert!(!Rung::Hexagon.serves(Role::Embedder));
assert!(!Rung::Hexagon.serves(Role::Denoiser));
assert!(!Rung::Hexagon.serves(Role::EyeClassifier));
assert!(Rung::Hexagon.serves(Role::Detector));
assert_eq!(Rung::Hexagon.form(Role::Detector), Form::Int8);
assert!(Rung::Hexagon.serves(Role::Denoiser));
assert_eq!(Rung::Hexagon.form(Role::Detector), Form::A16W8);
// A detector offered in f32 on a Hexagon device lands on the CPU.
let s = State {
config: Config::default(),
@@ -625,36 +684,57 @@ mod tests {
probing: false,
wanted: 0,
};
let on = |role, form| effective_rung(&s, Rung::Hexagon, role, form, engines::hash(b""));
assert_eq!(on(Role::Embedder, Form::F32), Rung::Cpu);
assert_eq!(on(Role::Detector, Form::F32), Rung::Cpu);
// A form other than the one the role wants is not the NPU's either:
// an int8 detector left over from an older install stays off it.
assert_eq!(on(Role::Detector, Form::Int8), Rung::Cpu);
// The wanted form whose context is not compiled yet: also the CPU.
assert_eq!(on(Role::Detector, Form::A16W8), Rung::Cpu);
}
/// The form each role gets on the Hexagon is the one measured to hold
/// its accuracy there (§1.5); a change to this table is a change to
/// what the tablet computes, and must come with a measurement.
#[test]
fn each_role_has_its_measured_form_on_the_hexagon() {
use Form::*;
for (role, form) in [
(Role::Detector, A16W8),
(Role::Landmarks, A16W8),
(Role::Segmenter, A16W16),
(Role::Scene, A16W16),
(Role::Inpainter, A16W16),
(Role::Denoiser, A16W16),
(Role::Keypoints, Int8),
(Role::Embedder, F32),
(Role::EyeClassifier, F32),
] {
assert_eq!(Rung::Hexagon.form(role), form, "{role:?}");
}
for rung in [
Rung::Cpu,
Rung::Cuda,
Rung::TensorRt,
Rung::MiGraphX,
Rung::CoreMl,
] {
assert_eq!(rung.form(Role::Detector), F32);
}
}
#[test]
fn a_form_lives_in_its_tagged_sibling() {
let canonical = Path::new("/m/scrfd_500m_640.onnx");
assert_eq!(form_sibling(canonical, Form::F32), canonical);
assert_eq!(
effective_rung(
&s,
Rung::Hexagon,
Role::Embedder,
Form::F32,
engines::hash(b"")
),
Rung::Cpu
form_sibling(canonical, Form::A16W8),
Path::new("/m/scrfd_500m_640.a16w8.onnx")
);
assert_eq!(
effective_rung(
&s,
Rung::Hexagon,
Role::Detector,
Form::F32,
engines::hash(b"")
),
Rung::Cpu
);
// An int8 detector whose context is not compiled yet: also the CPU.
assert_eq!(
effective_rung(
&s,
Rung::Hexagon,
Role::Detector,
Form::Int8,
engines::hash(b"")
),
Rung::Cpu
form_sibling(canonical, Form::Int8),
Path::new("/m/scrfd_500m_640.int8.onnx")
);
}
core/dr-inference-engine/src/probe.rs
+20 -19
View File
@@ -174,10 +174,9 @@ pub fn attempt<T>(cfg: &Config, what: &str, f: impl FnOnce() -> T) -> Result<T,
}
/// The smallest detector, or the smallest model of any role if there is
/// none. A ~2 MB detector is the cheapest real test of a provider, and the
/// detector is the role the int8 forms exist for — the eye classifiers are
/// smaller still, and a Hexagon probed with one would fail for want of a
/// form nobody ships.
/// none. A ~2 MB detector is the cheapest real test of a provider, and
/// every rung serves it — the eye classifiers are smaller still, but the
/// Hexagon does not take them, and a probe with one would fail it for that.
fn probe_model(cfg: &Config) -> Option<(Role, PathBuf)> {
let smallest = |want: Option<Role>| {
cfg.models
@@ -203,16 +202,14 @@ fn time_rung(
cfg: &Config,
) -> Result<(f64, Option<String>), String> {
let want = rung.form(role);
let path = match want {
Form::Int8 => {
let p = crate::int8_sibling(canonical);
if !p.is_file() {
return Err(format!("no int8 form of {}", canonical.display()));
}
p
}
Form::F32 => canonical.to_path_buf(),
};
let path = crate::form_sibling(canonical, want);
if want != Form::F32 && !path.is_file() {
return Err(format!(
"no {} form of {}",
want.file_tag().unwrap_or("f32"),
canonical.display()
));
}
let bytes = std::fs::read(&path).map_err(|e| e.to_string())?;
let started = Instant::now();
let mut session =
@@ -295,9 +292,9 @@ fn fingerprint(runtime: &Runtime, cfg: &Config) -> String {
},
device_identity(),
];
for (role, bytes) in &cfg.embedded {
for (role, form, bytes) in &cfg.embedded {
parts.push(format!(
"{role:?} embedded {:016x}",
"{role:?} embedded {form:?} {:016x}",
crate::engines::hash(bytes)
));
}
@@ -306,9 +303,13 @@ fn fingerprint(runtime: &Runtime, cfg: &Config) -> String {
.map(|b| crate::engines::hash(&b))
.unwrap_or(0);
parts.push(format!("{role:?} {hash:016x}"));
let int8 = crate::int8_sibling(path);
if let Ok(b) = std::fs::read(&int8) {
parts.push(format!("{role:?} int8 {:016x}", crate::engines::hash(&b)));
for form in [Form::Int8, Form::A16W8, Form::A16W16] {
if let Ok(b) = std::fs::read(crate::form_sibling(path, form)) {
parts.push(format!(
"{role:?} {form:?} {:016x}",
crate::engines::hash(&b)
));
}
}
}
parts.join("\n")