Add OpenVINO and WebGPU rungs to the inference ladder

OpenVINO is the Intel rung: the integrated or Arc GPU, fp16 for every
role but the embedder, a compiled program per model kept in a directory
per model, precision and runtime version. On the Iris Xe it beats ONNX
Runtime's CPU provider on every shipped model — scrfd_10g 23 ms against
58, the scene model 17 against 57, MI-GAN 57 against 330, a denoise tile
40 against 158.

WebGPU is the generic rung for a GPU no vendor rung covers. It was slower
than the CPU on the Iris Xe, the RTX 3050 and the Adreno, so it is on the
ladder for the GPUs it has not been timed on, behind the probe's clock.

MIGraphX's registration becomes one generic key/value helper that all
three share, with option names read from each runtime's own source.
This commit is contained in:
2026-10-04 21:00:15 -04:00
parent 555ec0efb3
commit 87c405eb46
4 changed files with 182 additions and 20 deletions
+24 -1
View File
@@ -48,12 +48,35 @@ pub fn context_path(cfg: &Config, bytes: &[u8]) -> PathBuf {
/// CoreML's own cache key leaves out the weights of a model loaded from
/// memory (`session::coreml`), and one per runtime version, which wrote it.
pub fn coreml_dir(cfg: &Config, bytes: &[u8]) -> PathBuf {
model_dir(cfg, "coreml", bytes)
}
/// Where OpenVINO compiles `bytes` to, at one precision. OpenVINO hashes
/// the model it is given, weights included, but a key that leaves out
/// what is being varied has cost a day before (CLAUDE.md, "Providers"),
/// and a directory per model and precision costs nothing: the precision
/// is a compile option, and the two forms are different programs.
pub fn openvino_dir(cfg: &Config, bytes: &[u8], fp16: bool) -> PathBuf {
model_dir(
cfg,
if fp16 {
"openvino/fp16"
} else {
"openvino/f32"
},
bytes,
)
}
/// `<cache>/<provider>/<runtime version>/<hash of the bytes>`: one per
/// model, and one per runtime version, which wrote it.
fn model_dir(cfg: &Config, provider: &str, bytes: &[u8]) -> PathBuf {
let runtime = match crate::api::runtime() {
crate::Runtime::OnnxRuntime { version, .. } => version,
crate::Runtime::Tract => "tract".into(),
};
cfg.cache_dir
.join("coreml")
.join(provider)
.join(runtime)
.join(format!("{:016x}", hash(bytes)))
}