Add OpenVINO and WebGPU rungs to the inference ladder
OpenVINO is the Intel rung: the integrated or Arc GPU, fp16 for every role but the embedder, a compiled program per model kept in a directory per model, precision and runtime version. On the Iris Xe it beats ONNX Runtime's CPU provider on every shipped model — scrfd_10g 23 ms against 58, the scene model 17 against 57, MI-GAN 57 against 330, a denoise tile 40 against 158. WebGPU is the generic rung for a GPU no vendor rung covers. It was slower than the CPU on the Iris Xe, the RTX 3050 and the Adreno, so it is on the ladder for the GPUs it has not been timed on, behind the probe's clock. MIGraphX's registration becomes one generic key/value helper that all three share, with option names read from each runtime's own source.
This commit is contained in:
@@ -48,12 +48,35 @@ pub fn context_path(cfg: &Config, bytes: &[u8]) -> PathBuf {
|
||||
/// CoreML's own cache key leaves out the weights of a model loaded from
|
||||
/// memory (`session::coreml`), and one per runtime version, which wrote it.
|
||||
pub fn coreml_dir(cfg: &Config, bytes: &[u8]) -> PathBuf {
|
||||
model_dir(cfg, "coreml", bytes)
|
||||
}
|
||||
|
||||
/// Where OpenVINO compiles `bytes` to, at one precision. OpenVINO hashes
|
||||
/// the model it is given, weights included, but a key that leaves out
|
||||
/// what is being varied has cost a day before (CLAUDE.md, "Providers"),
|
||||
/// and a directory per model and precision costs nothing: the precision
|
||||
/// is a compile option, and the two forms are different programs.
|
||||
pub fn openvino_dir(cfg: &Config, bytes: &[u8], fp16: bool) -> PathBuf {
|
||||
model_dir(
|
||||
cfg,
|
||||
if fp16 {
|
||||
"openvino/fp16"
|
||||
} else {
|
||||
"openvino/f32"
|
||||
},
|
||||
bytes,
|
||||
)
|
||||
}
|
||||
|
||||
/// `<cache>/<provider>/<runtime version>/<hash of the bytes>`: one per
|
||||
/// model, and one per runtime version, which wrote it.
|
||||
fn model_dir(cfg: &Config, provider: &str, bytes: &[u8]) -> PathBuf {
|
||||
let runtime = match crate::api::runtime() {
|
||||
crate::Runtime::OnnxRuntime { version, .. } => version,
|
||||
crate::Runtime::Tract => "tract".into(),
|
||||
};
|
||||
cfg.cache_dir
|
||||
.join("coreml")
|
||||
.join(provider)
|
||||
.join(runtime)
|
||||
.join(format!("{:016x}", hash(bytes)))
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user