jellytau/docs/build/ci-operations.md

# CI operations

How the pipeline is kept working: the builder image, the secrets it needs, the
gates that must stay required, and the things that only a human with access to
the Gitea instance can do.

CI is **Gitea Actions** (`.gitea/workflows/`) on `gitea.tourolle.paris`, not
GitHub.

## The workflows

| Workflow | Trigger | What it protects |
|---|---|---|
| [build-and-test.yml](../../.gitea/workflows/build-and-test.yml) | push/PR to `master` | Frontend + Rust gates, Android compile check, supply chain |
| [traceability-check.yml](../../.gitea/workflows/traceability-check.yml) | push/PR | Requirement coverage ratchet, dangling IDs |
| [build-release.yml](../../.gitea/workflows/build-release.yml) | tag `v*` | Builds, signs, publishes, and writes the update manifest |
| [publish-docs.yml](../../.gitea/workflows/publish-docs.yml) | push to `master` | Docs site on the `gitea-pages` branch |

## 🔴 CI installs no system tools

Every build, test and packaging **tool** lives in the Docker image the job runs
in. Never add `apt-get`, `rustup`, `sdkmanager`, or a `curl | tar -xz` of a
binary to a workflow step.

Fetching the project's *own declared dependencies* is not a toolchain install and
is fine: `bun install`, cargo pulling crates from the lockfile, `cargo deny`
fetching the RustSec advisory database. The distinction is tool versus data.

This rule has been broken twice, both times invisibly until something else
failed. `publish-docs.yml` downloaded mdBook from GitHub releases into
`/usr/local/bin` at job time — a hard dependency on GitHub's CDN being up
whenever docs were published. Both mdBook and the supply-chain tools are in the
image now.

## The builder image

`Dockerfile.builder` → `gitea.tourolle.paris/dtourolle/jellytau-builder`.
It carries: the pinned Rust toolchain plus rustfmt/clippy and the Android,
Windows-MSVC targets; bun and Node; the Android SDK/NDK and a local Gradle
distribution; Linux desktop and packaging deps (WebKitGTK, libmpv, rpm, NSIS,
cargo-xwin); and the tooling — `cargo-deny`, `cargo-cyclonedx`, `mdbook`.

Arch packages build in a separate `Dockerfile.arch`, because `makepkg` is
Arch-specific.

### Tags are pinned, and why

Workflows name an **immutable dated tag** (`:2026.08`), never `:latest`. While
every job said `:latest`, rebuilding the image silently changed what every build
compiled against — including a rebuild of an old release tag, which is the
opposite of reproducible.

`:latest` is still pushed alongside, for local `docker compose` runs and manual
pulls.

Date tags rather than per-commit SHA tags on purpose: the runner shares a 74 GB
disk with two other projects, and SHA-tagged images accumulated there until it
filled. Keep a couple of dated tags live and prune the rest.

### Changing the image

The order matters — CI breaks if the workflow lands before the image exists.

A caveat learned the hard way: the *trailing* layer is only fast for `cargo
install` tools. Adding an **apt** package invalidates the packaging layer, which
sits above the `cargo-xwin`/`cargo-deny` installs, so those recompile too — a
~20 minute rebuild rather than ~2.

```bash
# 1. Edit Dockerfile.builder. Put new tools in the TRAILING layer: it exists so
#    a tool change is a ~2 min rebuild instead of ~15.
# 2. Build and push, tagged with the new month:
./scripts/build-builder-image.sh 2026.09
# 3. Repoint every workflow at the new tag, in the same commit as whatever
#    needed the new tool:
sed -i 's|jellytau-builder:2026.08|jellytau-builder:2026.09|g' .gitea/workflows/*.yml
# 4. Verify the tools are actually in it:
docker run --rm gitea.tourolle.paris/dtourolle/jellytau-builder:2026.09 \
  -c "cargo deny --version; mdbook --version"
```

🔴 The Rust version is pinned in **two** places that must agree:
`RUST_VERSION` in `Dockerfile.builder` and `channel` in
`src-tauri/rust-toolchain.toml`. If they drift, rustup downloads the pinned
toolchain inside the job — a toolchain install in CI. Bump both, rebuild, push,
then merge.

## Tauri plugin versions are pinned in pairs

Every Tauri plugin exists twice: a Rust crate in `src-tauri/Cargo.toml` and an
npm package in `package.json`. **The Tauri CLI refuses to build when the two are
on different minor versions** — not a warning, a hard stop before compilation.

Both sides are therefore pinned *exactly* (`"2.8.0"`, not `"^2.8.0"`). A caret
range is what let them drift apart in the first place: `bun add` took the latest
npm package while cargo held an older crate, and nothing noticed until a release
build refused to start.

Nothing in `build-and-test.yml` runs `tauri build` — that happens only on a tag —
so this class of breakage used to be invisible until release day. The
`Check Tauri plugin versions match` step runs `tauri info`, which performs the
same comparison without building.

To upgrade a plugin, move **both** sides together and re-run that step. Expect
the Rust side to be the constraint: a newer plugin crate may pull a large
transitive upgrade (bumping `tauri-plugin-log` to 2.9.0 also moved `wry`,
`wasm-bindgen`, `web-sys` and `webkit2gtk`), which touches the webview and
therefore video playback. That is a change to make deliberately, with a full
build and a playback check — not one to slip into a release.

## AppImage needs more than the Rust toolchain

`linuxdeploy` (which Tauri downloads at build time to assemble the AppImage)
shells out to distro tools that a minimal server image does not have. It aborts
the whole bundle on the first one missing:

```
failed to bundle project: xdg-open binary not found
```

The image therefore carries `xdg-utils`, `desktop-file-utils` and `zsync`. This
is a class of failure that **cannot be caught by building locally**: a developer
machine is a desktop and has all three, so the AppImage builds there and fails in
CI. It cost one release build to find.

Tauri's AppImage bundler also downloads `linuxdeploy`, `AppRun` and two plugin
scripts from GitHub during the build. That is Tauri's behaviour, not ours, but it
means an AppImage build depends on GitHub being reachable from the runner.

## Secrets

Managed with the `tea` CLI (`tea actions secrets list`) or the repo settings UI.

| Secret | Used by | Notes |
|---|---|---|
| `ANDROID_KEYSTORE_BASE64` | release | Base64 of the release keystore |
| `ANDROID_KEYSTORE_PASSWORD` | release | |
| `ANDROID_KEY_ALIAS` | release | |
| `ANDROID_KEY_PASSWORD` | release | |
| `TAURI_SIGNING_PRIVATE_KEY` | release | minisign key for the desktop updater |
| `TAURI_SIGNING_PRIVATE_KEY_PASSWORD` | release | |
| `GITEA_TOKEN` | release, docs | PAT; falls back to the auto-provided token |

The updater keypair's public half is committed in `src-tauri/tauri.conf.json` —
that one is meant to be public; it is what clients verify against. The private
half exists in the Gitea secret and in the maintainer's local `.env` (which is
gitignored) and at `~/.tauri/jellytau.key`.

**Losing the private key means losing the ability to ship updates to installed
desktop clients**, because they will only accept payloads signed by the key
matching the public key they were built with. Recovering means generating a new
pair, shipping a build carrying the new public key, and telling everyone on an
older build to reinstall by hand. Back it up.

## Required status checks

Gitea → repo Settings → Branches → protect `master`, requiring:

- `Run Tests`
- `Android Compile Check`
- `Supply Chain`
- the traceability job

Without branch protection, every gate in this document is advisory: a push
straight to `master` lands whether or not CI is red. That is the state the repo
was in for its whole history before this was set up.

## The runner

One self-hosted runner, one ~74 GB disk shared with two other projects. It fills,
and when it does the symptoms are misleading: cargo dying mid-link, docker
refusing to pull, `actions/cache` quietly not saving — anything except an obvious
out-of-space error. Check the disk first.

There is deliberately no scheduled job watching this. On a single-slot runner a
daily job occupies the slot and pulls the builder image to run `df`, and `df`
inside a container does not reliably describe the host's disk anyway — it would
cost real build capacity to report a number that might be wrong. Check it by hand
on the runner:

```bash
df -h /
docker system df -v
```

When it does fill:

```bash
docker image prune -a
docker volume prune -a     # the -a matters: without it, NAMED volumes are kept,
                           # which is exactly how this filled up unnoticed
```

Never cache `src-tauri/target` — it is ~16 GB, and caching it under several keys
is what filled the disk at ~1.15 GB/day. The workflows cache only the cargo
registry index and `.crate` tarballs; cargo re-extracts `registry/src` for free.

## Release verification

The steps that catch a broken release before users do are in
[release-checklist.md](../release-checklist.md) — in particular the update path:
`latest.json` must be live on the `updater` branch, both platform entries must
carry a non-empty signature, and the previous release should be installed and
asked to update to the new one.

## Bus factor

The Gitea instance holds the canonical remote, the signing secrets, the container
registry and the CI runner. **It is not backed up as part of this repository, and
nothing in this repository can restore it.** That is the largest single risk to
the project — larger than any gate in this document — and the backup lives
outside it.