Add OpenVINO and WebGPU rungs to the inference ladder
OpenVINO is the Intel rung: the integrated or Arc GPU, fp16 for every role but the embedder, a compiled program per model kept in a directory per model, precision and runtime version. On the Iris Xe it beats ONNX Runtime's CPU provider on every shipped model — scrfd_10g 23 ms against 58, the scene model 17 against 57, MI-GAN 57 against 330, a denoise tile 40 against 158. WebGPU is the generic rung for a GPU no vendor rung covers. It was slower than the CPU on the Iris Xe, the RTX 3050 and the Adreno, so it is on the ladder for the GPUs it has not been timed on, behind the probe's clock. MIGraphX's registration becomes one generic key/value helper that all three share, with option names read from each runtime's own source.
This commit is contained in:
@@ -14,18 +14,27 @@ use crate::{api::Runtime, state, Cache, Config, Form, Role, Rung};
|
||||
|
||||
/// The rungs to try on this platform, best first, under the user's ceiling.
|
||||
fn ladder(ceiling: Option<Rung>) -> Vec<Rung> {
|
||||
// WebGPU is the generic rung (§2): it is reached only on a runtime that
|
||||
// carries it, which `api` loads where no vendor's runtime fits the
|
||||
// device, and kept only where it beats the CPU.
|
||||
#[cfg(target_os = "android")]
|
||||
let all = [Rung::Hexagon];
|
||||
let all = [Rung::Hexagon, Rung::WebGpu];
|
||||
// Unmeasured (§2 ⁵): it is on the ladder because the probe's clock and
|
||||
// `attempt` make a wrong guess cost one slow or failed probe, not a
|
||||
// slow or crashing app.
|
||||
#[cfg(target_os = "macos")]
|
||||
let all = [Rung::CoreMl];
|
||||
// A desktop has one vendor's GPU; the other vendor's providers are
|
||||
// "not enabled in this build" or a library that fails to load, and
|
||||
// either answer arrives in milliseconds.
|
||||
// A runtime carries one vendor's providers, chosen for this device's
|
||||
// GPU (`api`); the others are "not enabled in this build", and that
|
||||
// answer arrives in milliseconds.
|
||||
#[cfg(not(any(target_os = "android", target_os = "macos")))]
|
||||
let all = [Rung::TensorRt, Rung::Cuda, Rung::MiGraphX];
|
||||
let all = [
|
||||
Rung::TensorRt,
|
||||
Rung::Cuda,
|
||||
Rung::MiGraphX,
|
||||
Rung::OpenVino,
|
||||
Rung::WebGpu,
|
||||
];
|
||||
all.into_iter()
|
||||
.filter(|r| ceiling.is_none_or(|c| *r <= c))
|
||||
.collect()
|
||||
|
||||
Reference in New Issue
Block a user