Files
DarkRoom/core/dr-catalog/src/sync.rs
T
dtourolle 49b7bc2f9d Build the upload snapshot without the face crops instead of stripping them
Each sync pass spent 0.8-2.0 s of CPU and 1.0-5.4 s wall on the upload
snapshot of the reference catalog (24k images, 18,871 faces), ahead of the
rest of the pass. The upload itself had been crop-less since the crops
moved to the face shards. The cost was in how it got that way. The backup
API copied all 158 MB of the catalog, 96 MB of it the ~5 KB JPEG crop on
every faces row. Then `UPDATE faces SET crop = NULL` rewrote 18.9k rows
and freed their overflow chains, and VACUUM rebuilt the file again. That
wrote the catalog about three times over to upload 50 MB.

The snapshot is now built rather than copied. An empty file attaches the
catalog, creates each table from the catalog's own sqlite_master and fills
it with INSERT ... SELECT, with faces.crop selected as NULL. Indexes,
triggers and views follow, and user_version, application_id, page size and
the WAL header flag are carried over. It all runs in one transaction on the
snapshot's connection, so the catalog is read as of one moment and
concurrent writers are serialised, not raced, as the backup API did. The
build journal is in memory with synchronous off, because the file is
scratch that is rebuilt every pass and quick_check'd before upload. Foreign
keys are off on that connection. The bundled SQLite enables them, and then
a multi-row INSERT into images scans images for children of each new row
(shadowed_by is a self-reference with no index), which cost 1.2 s alone.

Measured on a .backup copy of the reference catalog with catalog_bench,
old and new binaries back to back on a loaded machine:
  before  best 1.0-5.4 s wall, 0.84-1.98 s cpu, 49.8 MB
  after   best 0.40-2.1 s wall, 0.39-0.96 s cpu, 50.4 MB
With the machine quiet the new build takes 0.31-0.43 s.

What a receiving device gets is unchanged. It is the same schema, the same
rows and a NULL crop, which is what 0.16.0 already uploads and merges. The
merge reads only a remote face's box and model (merge::match_faces) and
never writes a local crop. No device adopts a downloaded catalog as its
own, and a fresh one takes faces and crops from the shards. There is no
schema bump, so older builds still merge it. NFR-R2 backups keep using the
backup API and keep their crops.

Tests: the snapshot matches the catalog in schema, row counts, pragmas and
WAL header. A leftover file is replaced. Merging a crop-less snapshot
carries a confirmed name across by box and leaves the local crop
untouched, and does so idempotently.
2026-09-26 14:03:27 -04:00

723 lines
28 KiB
Rust

//! TRACES: FR-CAT-7 | FR-NC-9 | NFR-R1
//! Preparing the catalog file for upload, and taking in a remote one.
//!
//! # The hazard this module exists to handle
//!
//! A WAL-mode SQLite database is not one file. Committed transactions can live
//! in `catalog.sqlite-wal` with the main file lagging behind, so copying
//! `catalog.sqlite` alone uploads a **torn snapshot**: internally consistent as
//! of some older point, missing everything since. Worse, a naive copy taken
//! while a writer is mid-transaction can be structurally corrupt.
//!
//! So an upload never copies the live file. It runs a TRUNCATE checkpoint to
//! fold the WAL back into the main file, then uses SQLite's own backup API to
//! take a consistent snapshot — which serialises correctly against concurrent
//! writers rather than racing them.
//!
//! # What is actually synced
//!
//! Only the *user's judgements about their library* merge: collections, and the
//! keyword vocabulary with its assignments (see [`crate::merge`]). The rest of
//! the catalog is a *local index* of *local* storage — folder mtimes, cache
//! paths, job rows — and copying another device's version of those in would be
//! actively wrong. The remote file is read for those two and then discarded.
//!
//! This is why the catalog remains disposable in the ARCH §6.12 sense: nothing
//! here makes the local database authoritative for anything a rebuild could
//! not recover.
use std::path::{Path, PathBuf};
use rusqlite::Connection;
use crate::error::CatalogError;
use crate::merge::{self, MergeReport};
/// Schema name the downloaded remote catalog is attached under.
const REMOTE_SCHEMA: &str = "remote_cat";
/// Fold the WAL into the main database file.
///
/// TRUNCATE rather than PASSIVE: passive checkpointing gives up when a reader
/// holds the WAL open, which would leave recent commits out of the snapshot
/// without saying so.
pub fn checkpoint(conn: &Connection) -> Result<(), CatalogError> {
conn.pragma_update(None, "wal_checkpoint", "TRUNCATE")?;
Ok(())
}
/// Write a consistent snapshot of the catalog to `dest`, ready to upload,
/// without the face crops.
///
/// Built rather than copied. The snapshot is the *whole catalog* bar the
/// crops, uploaded on every sync and downloaded by every device. The crops are
/// most of the file (96 MB of a 158 MB reference catalog), and copying them in
/// only to delete them was most of the cost. A backup-API copy followed by
/// `UPDATE faces SET crop = NULL` and `VACUUM` wrote the file roughly three
/// times over to produce 50 MB (#71). So this creates the schema in an empty
/// file and copies every table into it with `crop` left NULL. That is one
/// pass, with nothing written that is not uploaded.
///
/// Consistency comes from doing the whole copy inside one transaction on the
/// snapshot's connection, which holds a single read snapshot of the source for
/// its duration. A writer committing meanwhile lands in the source's WAL and is
/// simply not seen, the same serialisation the backup API gave.
///
/// Crops are not lost by this: they travel in the face shards
/// ([`crate::face_shard::export_to_shards`]), which are written once and
/// downloaded once. Nothing reads a crop out of a merged remote catalog. The
/// merge reads a remote face's box and model to match it to a local one, and
/// no more. So leaving them out costs a receiving device nothing it would
/// otherwise have had. A device never adopts a downloaded catalog as its own,
/// so a fresh one gets its crops from the shards too.
///
/// The result must stay what every earlier build already merges: same schema,
/// same `user_version`, same page size and the same WAL flag in the header.
/// The `the_snapshot_*` tests pin those.
pub fn snapshot_for_upload(conn: &Connection, dest: &Path) -> Result<(), CatalogError> {
// The source is read through a second connection, attached to the
// snapshot's, so it needs to be a file. Every catalog is one.
let source = conn
.path()
.filter(|p| !p.is_empty())
.map(PathBuf::from)
.ok_or_else(|| CatalogError::Io("the catalog to snapshot has no file".into()))?;
// Not needed for consistency, since the read transaction below sees the
// WAL, but it keeps the live WAL from growing across syncs, as before.
checkpoint(conn)?;
// A leftover from a pass that died mid-build would otherwise be built on.
for stale in [
dest.to_path_buf(),
sidecar_of(dest, "-wal"),
sidecar_of(dest, "-journal"),
sidecar_of(dest, "-shm"),
] {
match std::fs::remove_file(&stale) {
Ok(()) => {}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => return Err(CatalogError::Io(format!("{}: {e}", stale.display()))),
}
}
let out = Connection::open(dest)?;
build_snapshot(conn, &source, &out)?;
verify_snapshot(&out)?;
Ok(())
}
/// `catalog.sqlite` + `-wal` → `catalog.sqlite-wal`.
fn sidecar_of(path: &Path, suffix: &str) -> PathBuf {
let mut s = path.as_os_str().to_owned();
s.push(suffix);
PathBuf::from(s)
}
/// Schema name the source catalog is attached under while a snapshot is built.
const SOURCE_SCHEMA: &str = "snap_src";
/// The body of [`snapshot_for_upload`]: fill the empty database `out` from
/// the catalog at `source`.
fn build_snapshot(conn: &Connection, source: &Path, out: &Connection) -> Result<(), CatalogError> {
// Settings that only take on an empty file, copied from the source so the
// result is the file a backup would have been.
let page_size: i64 = conn.query_row("PRAGMA main.page_size", [], |r| r.get(0))?;
let auto_vacuum: i64 = conn.query_row("PRAGMA main.auto_vacuum", [], |r| r.get(0))?;
out.pragma_update(None, "page_size", page_size)?;
out.pragma_update(None, "auto_vacuum", auto_vacuum)?;
// A scratch file, rebuilt whole on every pass and checked before upload:
// durability during the build buys nothing. MEMORY rather than OFF keeps
// ROLLBACK defined on the failure path.
out.pragma_update(None, "journal_mode", "MEMORY")?;
out.pragma_update(None, "synchronous", "OFF")?;
// The rows were checked when they were written, and the copy has them
// all by the end. The bundled SQLite turns foreign keys on by default,
// and with them a multi-row INSERT into `images` scans `images` for
// children of every row it adds (`shadowed_by` refers to the same table
// and has no index): 1.2 s of a 1.7 s snapshot on 24k images.
out.pragma_update(None, "foreign_keys", false)?;
// Bound as a parameter, so a path containing a quote cannot break out.
out.execute(
&format!("ATTACH DATABASE ?1 AS {SOURCE_SCHEMA}"),
[source.to_string_lossy().as_ref()],
)?;
let result = copy_schema_and_rows(out);
if let Err(e) = out.execute(&format!("DETACH DATABASE {SOURCE_SCHEMA}"), []) {
log::warn!("failed to detach the catalog from its snapshot: {e}");
}
result?;
// Last, and outside any transaction, which is the only place it can be
// set: the header says WAL, as every snapshot uploaded so far has.
out.pragma_update(None, "journal_mode", "WAL")?;
Ok(())
}
fn copy_schema_and_rows(out: &Connection) -> Result<(), CatalogError> {
let tx = out.unchecked_transaction()?;
// The first read of the source opens its read snapshot. Everything from
// here, schema included, is as of that one moment.
let objects: Vec<(String, String, String)> = {
let mut stmt = tx.prepare(&format!(
"SELECT type, name, sql FROM {SOURCE_SCHEMA}.sqlite_master
WHERE sql IS NOT NULL AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
ORDER BY rowid"
))?;
let rows = stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))?;
rows.collect::<Result<_, _>>()?
};
let user_version: i64 =
tx.query_row(&format!("PRAGMA {SOURCE_SCHEMA}.user_version"), [], |r| {
r.get(0)
})?;
let application_id: i64 =
tx.query_row(&format!("PRAGMA {SOURCE_SCHEMA}.application_id"), [], |r| {
r.get(0)
})?;
// Tables and their rows first, then indexes, triggers and views, so that
// an index is built once over the data rather than maintained per row,
// and no trigger fires on the copy. Foreign keys are off on this
// connection, so the order tables are filled in does not matter.
for (_, name, sql) in objects.iter().filter(|(k, _, _)| k == "table") {
// Verbatim: an unqualified CREATE lands in `main`, the snapshot.
tx.execute_batch(sql)?;
let columns: Vec<String> = {
let mut stmt = tx.prepare("SELECT name FROM pragma_table_info(?1, 'main')")?;
let rows = stmt.query_map([name], |r| r.get::<_, String>(0))?;
rows.collect::<Result<_, _>>()?
};
let select = columns
.iter()
.map(|c| {
if name == "faces" && c == "crop" {
"NULL".to_string()
} else {
quote_ident(c)
}
})
.collect::<Vec<_>>()
.join(", ");
let insert = columns
.iter()
.map(|c| quote_ident(c))
.collect::<Vec<_>>()
.join(", ");
let table = quote_ident(name);
tx.execute(
&format!(
"INSERT INTO main.{table} ({insert})
SELECT {select} FROM {SOURCE_SCHEMA}.{table}"
),
[],
)?;
}
// AUTOINCREMENT's counters live in a table the filter above skips; the
// CREATE of such a table makes an empty one here.
let has_sequence: bool = tx.query_row(
&format!(
"SELECT EXISTS(SELECT 1 FROM {SOURCE_SCHEMA}.sqlite_master
WHERE name = 'sqlite_sequence')"
),
[],
|r| r.get(0),
)?;
if has_sequence {
tx.execute_batch(&format!(
"DELETE FROM main.sqlite_sequence;
INSERT INTO main.sqlite_sequence SELECT * FROM {SOURCE_SCHEMA}.sqlite_sequence;"
))?;
}
for (_, _, sql) in objects.iter().filter(|(k, _, _)| k != "table") {
tx.execute_batch(sql)?;
}
tx.pragma_update(None, "user_version", user_version)?;
tx.pragma_update(None, "application_id", application_id)?;
tx.commit()?;
Ok(())
}
/// `name` as an SQL identifier, whatever it contains.
fn quote_ident(name: &str) -> String {
format!("\"{}\"", name.replace('"', "\"\""))
}
/// TRACES: NFR-R2
/// Refuse to hand over a snapshot that will not pass `quick_check`.
///
/// The upload is the copy every other device merges from, and a damaged one
/// costs far more than the check: each device downloads it, fails, and — for
/// a week, once — declines to push over it. `quick_check` reads every page
/// but skips index verification, which is the affordable version of "is this
/// a database" on a 40 MB file that has just been written and is still in the
/// page cache. A failure here is [`CatalogError::Corrupt`], the same thing a
/// receiving device would have said, so the sync reports it the same way.
fn verify_snapshot(snapshot: &Connection) -> Result<(), CatalogError> {
let verdict: String = snapshot.query_row("PRAGMA quick_check", [], |r| r.get(0))?;
if verdict == "ok" {
Ok(())
} else {
Err(CatalogError::Corrupt {
detail: format!("the snapshot for upload failed quick_check: {verdict}"),
})
}
}
/// Checkpoint, then copy the whole database to `dest`, and hand back the
/// connection to the copy.
///
/// What [`crate::recovery`] takes its NFR-R2 backups with. A backup is the
/// file the user may have to *live on*, so it keeps the face crops that
/// [`snapshot_for_upload`] leaves out, and a byte-for-byte page copy is the
/// right tool. Keeping it here beside the upload keeps the WAL discipline in
/// one place: a backup taken with `fs::copy` would be the torn snapshot this
/// module's header exists to warn about.
pub(crate) fn copy_to(conn: &Connection, dest: &Path) -> Result<Connection, CatalogError> {
checkpoint(conn)?;
let mut out = Connection::open(dest)?;
let backup = rusqlite::backup::Backup::new(conn, &mut out)?;
// SQLite's own "copy everything" sentinel is -1, but rusqlite asserts a
// positive page count, so ask for more pages than a catalog will ever
// have. The effect is the same: one step, no interleaved writers, no
// progress callback. A 50k-image catalog is tens of megabytes.
backup.run_to_completion(i32::MAX, std::time::Duration::ZERO, None)?;
drop(backup);
Ok(out)
}
/// Whether a downloaded remote catalog is worth merging.
///
/// Cheap guard before attaching: a remote written by a newer build may contain
/// tables and columns this one cannot read, and attempting the merge would
/// fail mid-transaction rather than declining cleanly.
pub fn remote_is_mergeable(remote: &Path) -> Result<bool, CatalogError> {
let conn = Connection::open_with_flags(
remote,
rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY | rusqlite::OpenFlags::SQLITE_OPEN_NO_MUTEX,
)?;
let v: i64 = conn.query_row("PRAGMA user_version", [], |r| r.get(0))?;
Ok(v <= crate::schema::SCHEMA_VERSION)
}
/// Attach a downloaded remote catalog, merge its collections, detach.
///
/// The remote file is opened **read-only** — this device never writes to
/// another device's catalog, it only reads collections out of it.
pub fn merge_remote(conn: &Connection, remote: &Path) -> Result<MergeReport, CatalogError> {
if !remote_is_mergeable(remote)? {
return Err(CatalogError::SchemaTooNew {
found: -1,
supported: crate::schema::SCHEMA_VERSION,
});
}
// Path binds as a parameter; ATTACH accepts one, so a path containing a
// quote cannot break out into SQL.
conn.execute(
&format!("ATTACH DATABASE ?1 AS {REMOTE_SCHEMA}"),
[remote.to_string_lossy().as_ref()],
)?;
let result = merge::merge_all(conn);
// Detach even if the merge failed, or the next attempt errors with
// "database remote_cat is already in use".
let detach = conn.execute(&format!("DETACH DATABASE {REMOTE_SCHEMA}"), []);
if let Err(e) = detach {
log::warn!("failed to detach remote catalog: {e}");
}
result
}
/// Where the catalog snapshot and the downloaded remote live.
///
/// Both are transient working files, not the catalog itself, so they belong in
/// the cache directory rather than beside the live database.
#[derive(Debug, Clone)]
pub struct SyncPaths {
pub upload_snapshot: PathBuf,
pub downloaded_remote: PathBuf,
}
impl SyncPaths {
pub fn in_dir(cache_dir: &Path) -> Self {
SyncPaths {
upload_snapshot: cache_dir.join("catalog-upload.sqlite"),
downloaded_remote: cache_dir.join("catalog-remote.sqlite"),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::schema;
fn seeded(path: &Path) -> Connection {
let c = Connection::open(path).unwrap();
schema::configure(&c).unwrap();
schema::migrate(&c).unwrap();
c
}
#[test]
fn snapshot_captures_committed_data() {
let dir = tempdir();
let live = dir.join("catalog.sqlite");
let snap = dir.join("snap.sqlite");
let c = seeded(&live);
c.execute(
"INSERT INTO collections(uuid, name, kind, created, revision, modified)
VALUES ('u1', 'Iceland', 0, 0, 1, 1)",
[],
)
.unwrap();
snapshot_for_upload(&c, &snap).unwrap();
// The snapshot must hold the row even though it was written after the
// database was created — the torn-file failure this guards against.
let s = Connection::open(&snap).unwrap();
let name: String = s
.query_row("SELECT name FROM collections", [], |r| r.get(0))
.unwrap();
assert_eq!(name, "Iceland");
}
#[test]
fn a_remote_from_a_newer_build_is_declined_not_attempted() {
let dir = tempdir();
let remote = dir.join("remote.sqlite");
let r = seeded(&remote);
r.pragma_update(None, "user_version", schema::SCHEMA_VERSION + 1)
.unwrap();
drop(r);
assert!(!remote_is_mergeable(&remote).unwrap());
let local = seeded(&dir.join("local.sqlite"));
assert!(matches!(
merge_remote(&local, &remote),
Err(CatalogError::SchemaTooNew { .. })
));
}
#[test]
fn merge_remote_round_trips_a_collection() {
let dir = tempdir();
let remote_path = dir.join("remote.sqlite");
{
let r = seeded(&remote_path);
r.execute(
"INSERT INTO collections(uuid, name, kind, created, revision, modified)
VALUES ('u-remote', 'Portugal', 0, 0, 1, 1)",
[],
)
.unwrap();
checkpoint(&r).unwrap();
}
let local = seeded(&dir.join("local.sqlite"));
local
.execute(
"INSERT INTO collections(uuid, name, kind, created, revision, modified)
VALUES ('u-local', 'Iceland', 0, 0, 1, 1)",
[],
)
.unwrap();
let report = merge_remote(&local, &remote_path).unwrap();
assert_eq!(report.inserted, 1);
let n: i64 = local
.query_row("SELECT count(*) FROM collections", [], |r| r.get(0))
.unwrap();
assert_eq!(n, 2);
}
#[test]
fn the_remote_can_be_merged_twice_without_attach_conflict() {
// Detach must happen even on the failure path, or the second attempt
// errors with "database remote_cat is already in use".
let dir = tempdir();
let remote_path = dir.join("remote.sqlite");
{
let r = seeded(&remote_path);
r.execute(
"INSERT INTO collections(uuid, name, kind, created, revision, modified)
VALUES ('u-remote', 'Portugal', 0, 0, 1, 1)",
[],
)
.unwrap();
checkpoint(&r).unwrap();
}
let local = seeded(&dir.join("local.sqlite"));
merge_remote(&local, &remote_path).unwrap();
let second = merge_remote(&local, &remote_path).unwrap();
assert!(!second.local_changed());
}
/// A scratch directory that cleans up with the test.
fn tempdir() -> PathBuf {
let base = std::env::temp_dir().join(format!(
"dr-catalog-test-{}-{:?}",
std::process::id(),
std::thread::current().id()
));
let _ = std::fs::remove_dir_all(&base);
std::fs::create_dir_all(&base).unwrap();
base
}
/// The whole reason crops live in the shards: a snapshot is uploaded whole,
/// on every sync, to every device.
#[test]
fn the_snapshot_carries_no_face_crops() {
let dir = tempdir();
let live = dir.join("catalog.sqlite");
let snap = dir.join("snap.sqlite");
let c = seeded(&live);
c.execute(
"INSERT OR IGNORE INTO roots(id, kind, label) VALUES (1, 'local', 'lib')",
[],
)
.unwrap();
c.execute(
"INSERT INTO images(id, root_id, source_ref, added_at) VALUES (1, 1, 'a.CR3', 0)",
[],
)
.unwrap();
c.execute(
"INSERT INTO faces
(image_id, x, y, w, h, landmarks, detector_confidence, embedding,
crop_px, model_id, detected_at, crop)
VALUES (1, 0.1, 0.1, 0.2, 0.2, X'00', 0.9, X'00', 180.0, 'm', 0, ?1)",
[vec![7u8; 4096]],
)
.unwrap();
snapshot_for_upload(&c, &snap).unwrap();
let out = Connection::open(&snap).unwrap();
let crops: i64 = out
.query_row(
"SELECT COUNT(*) FROM faces WHERE crop IS NOT NULL",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(crops, 0, "the snapshot still carries face crops");
// The face itself must still be there — only the pixels are dropped.
let faces: i64 = out
.query_row("SELECT COUNT(*) FROM faces", [], |r| r.get(0))
.unwrap();
assert_eq!(faces, 1);
// And the local catalog keeps its crop: this strips the copy, never
// the original.
let kept: i64 = c
.query_row(
"SELECT COUNT(*) FROM faces WHERE crop IS NOT NULL",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(kept, 1, "stripping the snapshot damaged the live catalog");
}
/// Device-side setup for the snapshot tests: an image both devices know by
/// its cross-device file id, and one face on it carrying `crop`.
fn with_a_face(c: &Connection, face_id: i64, x: f64, crop: &[u8]) {
c.execute(
"INSERT OR IGNORE INTO roots(id, kind, label) VALUES (1, 'local', 'lib')",
[],
)
.unwrap();
c.execute(
"INSERT INTO images(id, root_id, source_ref, added_at) VALUES (1, 1, 'a.CR3', 0)",
[],
)
.unwrap();
c.execute("INSERT INTO remote(image_id, file_id) VALUES (1, 5000)", [])
.unwrap();
c.execute(
"INSERT INTO faces
(id, image_id, x, y, w, h, landmarks, detector_confidence, embedding,
crop_px, model_id, detected_at, crop)
VALUES (?1, 1, ?2, 0.2, 0.2, 0.2, X'00', 0.9, X'00', 150.0, 'w600k_mbf', 0, ?3)",
rusqlite::params![face_id, x, crop],
)
.unwrap();
}
fn crop_of(c: &Connection, face_id: i64) -> Option<Vec<u8>> {
c.query_row("SELECT crop FROM faces WHERE id = ?1", [face_id], |r| {
r.get(0)
})
.unwrap()
}
/// The snapshot is built table by table rather than copied, so what has to
/// hold is that it is still the same database bar the crops: every table,
/// index and row, and the header fields an older build checks before it
/// will merge (`user_version`) or open it the way it always has (the WAL
/// flag and page size a backup-API copy carried).
#[test]
fn the_snapshot_is_the_catalog_bar_the_crops() {
let dir = tempdir();
let live = dir.join("catalog.sqlite");
let snap = dir.join("snap.sqlite");
let c = seeded(&live);
with_a_face(&c, 7, 0.3, &[7u8; 4096]);
c.execute(
"INSERT INTO collections(uuid, name, kind, created, revision, modified)
VALUES ('u1', 'Iceland', 0, 0, 1, 1)",
[],
)
.unwrap();
// Created on first use rather than by a migration: the copy must not
// depend on the migrations knowing every table.
crate::duplicates::ensure_probe_table(&c).unwrap();
snapshot_for_upload(&c, &snap).unwrap();
let out = Connection::open(&snap).unwrap();
let objects = |conn: &Connection| -> Vec<(String, String)> {
let mut stmt = conn
.prepare("SELECT type, name FROM sqlite_master ORDER BY type, name")
.unwrap();
let rows = stmt
.query_map([], |r| Ok((r.get(0).unwrap(), r.get(1).unwrap())))
.unwrap();
rows.map(Result::unwrap).collect()
};
assert_eq!(objects(&out), objects(&c), "the snapshot's schema differs");
for (kind, table) in objects(&c) {
if kind != "table" {
continue;
}
let count = |conn: &Connection| -> i64 {
conn.query_row(&format!("SELECT COUNT(*) FROM \"{table}\""), [], |r| {
r.get(0)
})
.unwrap()
};
assert_eq!(count(&out), count(&c), "rows differ in {table}");
}
for pragma in [
"user_version",
"application_id",
"page_size",
"journal_mode",
] {
let read = |conn: &Connection| -> String {
conn.query_row(&format!("PRAGMA {pragma}"), [], |r| {
r.get::<_, rusqlite::types::Value>(0)
})
.map(|v| format!("{v:?}"))
.unwrap()
};
assert_eq!(read(&out), read(&c), "{pragma} differs");
}
drop(out);
// Bytes 18 and 19 of the header are 2 for a WAL database, which is
// what every snapshot uploaded before this one said.
let header = std::fs::read(&snap).unwrap();
assert_eq!(&header[18..20], &[2, 2], "the snapshot is not WAL-flagged");
// The face is there; its pixels are not.
let out = Connection::open(&snap).unwrap();
assert_eq!(crop_of(&out, 7), None);
}
/// A pass that died mid-build leaves a file behind; the next one must
/// build afresh rather than on top of it.
#[test]
fn a_leftover_snapshot_is_replaced_not_built_on() {
let dir = tempdir();
let live = dir.join("catalog.sqlite");
let snap = dir.join("snap.sqlite");
let c = seeded(&live);
std::fs::write(&snap, b"not a database").unwrap();
snapshot_for_upload(&c, &snap).unwrap();
// And twice over a good one, which is the steady state.
snapshot_for_upload(&c, &snap).unwrap();
let out = Connection::open(&snap).unwrap();
let v: i64 = out
.query_row("PRAGMA user_version", [], |r| r.get(0))
.unwrap();
assert_eq!(v, schema::SCHEMA_VERSION);
}
/// The merge side of a crop-less snapshot: the other device's names still
/// cross over — the match is by box, not by pixels — and this device's
/// own crop is left exactly as it was, never replaced by the snapshot's
/// NULL.
#[test]
fn merging_a_crop_less_snapshot_keeps_local_crops_and_takes_the_names() {
let dir = tempdir();
let snap = dir.join("snap.sqlite");
let desktop = seeded(&dir.join("desktop.sqlite"));
with_a_face(&desktop, 42, 0.31, &[1u8; 3000]);
desktop
.execute(
"INSERT INTO people(id, uuid, name, ignored, created, revision, modified)
VALUES (3, 'u-anna', 'Anna', 0, 0, 1, 1)",
[],
)
.unwrap();
desktop
.execute(
"INSERT INTO face_person(face_id, person_id, probability, confirmed)
VALUES (42, 3, 0.9, 1)",
[],
)
.unwrap();
snapshot_for_upload(&desktop, &snap).unwrap();
// The tablet found the same face itself, under its own row id, and has
// its own crop of it — from its own detection or from the shards.
let tablet = seeded(&dir.join("tablet.sqlite"));
let mine = vec![9u8; 2500];
with_a_face(&tablet, 7, 0.30, &mine);
let report = merge_remote(&tablet, &snap).unwrap();
assert_eq!(report.people_inserted, 1);
assert_eq!(report.faces_assigned, 1);
let named: (String, bool) = tablet
.query_row(
"SELECT p.name, fp.confirmed
FROM face_person fp JOIN people p ON p.id = fp.person_id
WHERE fp.face_id = 7",
[],
|r| Ok((r.get(0)?, r.get(1)?)),
)
.unwrap();
assert_eq!(named, ("Anna".to_string(), true));
assert_eq!(
crop_of(&tablet, 7),
Some(mine),
"the merge touched a local crop"
);
// Idempotent over a crop-less remote too.
let again = merge_remote(&tablet, &snap).unwrap();
assert!(!again.local_changed());
}
}