Warm a rung up before the probe times it
One warm-up run and the median of three: on the Iris Xe the OpenVINO rung lost to the CPU on the smallest detector in two probes of three, because an idle integrated GPU takes a few runs to raise its clock — warm, it is 5.8 ms against 9.5. Three warm-ups and the median of seven took it in five probes of five (5.6–7.2 ms against 7.7–13.1). The extra runs cost tens of milliseconds, once per fingerprint.
This commit is contained in:
@@ -255,9 +255,14 @@ fn probe_model(cfg: &Config) -> Option<(Role, PathBuf)> {
|
|||||||
smallest(Some(Role::Detector)).or_else(|| smallest(None))
|
smallest(Some(Role::Detector)).or_else(|| smallest(None))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Build, run once for the engine, then time three runs; the median in
|
/// Build, warm up, then time seven runs; the median in milliseconds and,
|
||||||
/// milliseconds and, for a compiling rung, the cache key of the engine this
|
/// for a compiling rung, the cache key of the engine this just built.
|
||||||
/// just built.
|
///
|
||||||
|
/// Three warm-ups, not one: an idle integrated GPU takes a few runs to
|
||||||
|
/// raise its clock. With one, the Iris Xe's OpenVINO lost to the CPU on
|
||||||
|
/// the smallest detector in two probes of three, where warm it is 5.8 ms
|
||||||
|
/// against 9.5 (§1.6). The smallest detector is a GPU's worst case; the
|
||||||
|
/// clock must not also be.
|
||||||
fn time_rung(
|
fn time_rung(
|
||||||
rung: Rung,
|
rung: Rung,
|
||||||
role: Role,
|
role: Role,
|
||||||
@@ -315,11 +320,15 @@ fn time_rung(
|
|||||||
.map_err(|e| e.to_string())?;
|
.map_err(|e| e.to_string())?;
|
||||||
Ok(t.elapsed().as_secs_f64() * 1e3)
|
Ok(t.elapsed().as_secs_f64() * 1e3)
|
||||||
};
|
};
|
||||||
run(&mut session)?;
|
for _ in 0..3 {
|
||||||
let mut times = [run(&mut session)?, run(&mut session)?, run(&mut session)?];
|
run(&mut session)?;
|
||||||
|
}
|
||||||
|
let mut times = (0..7)
|
||||||
|
.map(|_| run(&mut session))
|
||||||
|
.collect::<Result<Vec<_>, _>>()?;
|
||||||
times.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
times.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||||
let key = rung.compiles().then(|| crate::engines::key(rung, &bytes));
|
let key = rung.compiles().then(|| crate::engines::key(rung, &bytes));
|
||||||
Ok((times[1], key))
|
Ok((times[times.len() / 2], key))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The part of a provider's error a person can act on. ONNX Runtime's
|
/// The part of a provider's error a person can act on. ONNX Runtime's
|
||||||
|
|||||||
Reference in New Issue
Block a user