dtourolle
05508741af
The desktop names where a package may have put libonnxruntime — an override variable, beside the executable, the package's own library directory, the Flatpak prefix, the system library directory — and Android points at the APK's native library directory, which is also what Qualcomm's DSP loader must be told for the Hexagon skel. Android starts the engine at the end of the model unpack rather than at launch, because the probe fingerprints the model files and a first launch has none until then. The About panel gains an Inference row beside Graphics, re-read every two seconds while the probe runs and engines land, and faces.model_id carries the detector's form: an int8 detector finds a different set of faces and is a different population (docs/inference.md §7). A low-memory signal drops every idle session with the GPU caches. The APK assembly bundles ONNX Runtime and the Qualcomm HTP libraries from Maven, fetched by tools/fetch-android-runtime.sh with their published checksums; RUNTIME_DIR=none builds the tract-only APK, which is a slower app and not a broken one. The desktop packages carry no runtime yet. Two probe fixes from the first desktop run: the floor must not be built with CPU fallback disabled, and a versioned libonnxruntime.so is a runtime too. On the reference desktop the probe now loads ONNX Runtime 1.30, measures 30 ms on the CPU provider, and selects TensorRT at 1.5 ms.
113 lines
5.0 KiB
Rust
113 lines
5.0 KiB
Rust
//! TRACES: FR-DEV-3i
|
||
//! Region segmentation for local masking (S15, docs/segmentation.md).
|
||
//!
|
||
//! Local adjustments need to know where the image's regions are before they
|
||
//! can snap a mask to one. This crate is that map, and it is deliberately
|
||
//! **device-free**: the watershed's pixel passes live in `dr-gpu` because they
|
||
//! are shaders, and everything that reasons about *regions* rather than
|
||
//! *pixels* lives here, where it can be tested on hand-built inputs with no
|
||
//! adapter present (ARCH §6.5a).
|
||
//!
|
||
//! # The three arms
|
||
//!
|
||
//! [`hierarchy`] is **arm A** — a watershed over-segments the image and the
|
||
//! recorded merge order becomes a granularity ladder. Deterministic, needs no
|
||
//! model, works on any picture, and knows nothing about what anything *is*.
|
||
//!
|
||
//! [`semantic`] is **arm B** — a YOLO instance-segmentation model naming the
|
||
//! subjects it recognises. Knows what things are, and is vague about exactly
|
||
//! where their edges fall (its prototypes are quarter-resolution).
|
||
//!
|
||
//! [`prior`] is **arm C**, and it is the one that ships. Arm B's instances
|
||
//! *re-weight* arm A's merge order, so coarse levels of the ladder line up
|
||
//! with real objects while every boundary stays exactly where the watershed
|
||
//! put it. The model contributes what it is good at — knowing what things are
|
||
//! — and the watershed contributes what it is good at, which is knowing where
|
||
//! the edge is, to the pixel, at every scale.
|
||
//!
|
||
//! That combination is also what repairs the vocabulary problem. The shipped
|
||
//! model is COCO-trained, so it recognises subjects and has no class for sky,
|
||
//! foliage or wall (`models/LICENCE.md`). Selecting those falls to arm A,
|
||
//! which never needed a vocabulary to begin with.
|
||
//!
|
||
//! # And [`scene`], which is not one of the arms
|
||
//!
|
||
//! The three arms all serve *local* adjustment: they exist so a mask can be
|
||
//! snapped to one region of the picture. [`scene`] serves the opposite move —
|
||
//! one grade applied to every pixel of a category at once, sky or foliage or
|
||
//! water — and reads a second, ADE20K-trained model to do it. It shares this
|
||
//! crate because it shares the runtime and the letterbox, not because it is
|
||
//! another way of doing the same thing.
|
||
//!
|
||
//! [`refine`] is what the scene model needs and the instance model does not.
|
||
//! A category's weights come off an 80×80 grid, so one cell is twenty pixels
|
||
//! of a 1600px proxy and anything smaller than that — a flag in the sky, a
|
||
//! chimney, a bare branch — is averaged into whatever surrounds it. There is
|
||
//! no tiling answer here the way there is for an instance, because a category
|
||
//! has no bounding box to tile over. So the fix is the same one arm C makes:
|
||
//! the model says *what*, and the photograph's own pixels say *which* of them
|
||
//! belong to it.
|
||
//!
|
||
//! It shares no code with the arms and it is not a fourth one — it sharpens a
|
||
//! mask that already exists rather than proposing regions — but it is built on
|
||
//! [`distance`] for the same reason arm C is built on the watershed, which is
|
||
//! that the useful question is always "how far inside am I".
|
||
|
||
pub mod distance;
|
||
pub mod hierarchy;
|
||
pub mod prior;
|
||
pub mod refine;
|
||
#[cfg(feature = "semantic")]
|
||
pub mod scene;
|
||
#[cfg(feature = "semantic")]
|
||
pub mod semantic;
|
||
|
||
pub use distance::{signed_distance, Falloff, Morphology, Shaped};
|
||
pub use hierarchy::{Edge, Merge, MergeTree, RegionField};
|
||
pub use prior::{Membership, PriorOptions};
|
||
pub use refine::{
|
||
refine_category, RefineOptions, Refined, Refinement, SkipReason, STRICTNESS_DEFAULT,
|
||
STRICTNESS_MAX, STRICTNESS_OFF,
|
||
};
|
||
#[cfg(feature = "semantic")]
|
||
pub use scene::{Category, Scene, SceneModel};
|
||
#[cfg(feature = "embedded-model")]
|
||
pub use semantic::embedded_model_bytes;
|
||
#[cfg(feature = "semantic")]
|
||
pub use semantic::{Instance, SemanticModel, SemanticOptions, Tiling};
|
||
|
||
/// What can go wrong between an image and a region map.
|
||
#[derive(Debug, thiserror::Error)]
|
||
pub enum SegmentError {
|
||
#[error("could not read model file: {0}")]
|
||
ModelRead(#[source] std::io::Error),
|
||
|
||
#[cfg(feature = "semantic")]
|
||
#[error("inference failed: {0}")]
|
||
Inference(#[source] ort::Error),
|
||
|
||
#[error("image buffer is {got} floats, expected {expected} (RGB, three per pixel)")]
|
||
ImageShape { expected: usize, got: usize },
|
||
|
||
/// The graph produced something the decoder does not recognise — a
|
||
/// different model, or a different export of the same one.
|
||
#[error("model output '{0}' did not have the expected shape")]
|
||
OutputShape(&'static str),
|
||
|
||
/// `models/scene/categories.txt` and the model disagree, or the descriptor
|
||
/// is malformed. Its own variant rather than a parse error because every
|
||
/// case carries a specific sentence about what to fix.
|
||
#[error("category descriptor: {0}")]
|
||
CategoryDescriptor(String),
|
||
}
|
||
|
||
#[cfg(feature = "semantic")]
|
||
impl From<dr_inference_engine::Error> for SegmentError {
|
||
fn from(e: dr_inference_engine::Error) -> Self {
|
||
match e {
|
||
dr_inference_engine::Error::Inference(e) => SegmentError::Inference(e),
|
||
dr_inference_engine::Error::Io(e) => SegmentError::ModelRead(e),
|
||
}
|
||
}
|
||
}
|