dtourolle
fb72bf3005
The commit that removed .gitea/workflows/runner-health.yml was amended with a git add whose pathspec named the already-deleted file, so the add aborted and only the deletion was staged -- leaving ci-operations.md still describing a scheduled job that no longer exists. The runner section now says what to check by hand and why there is no job: on a single-slot runner a daily job takes the slot and pulls the builder image to run df, and df inside a container does not reliably describe the host disk.
166 lines
7.0 KiB
Markdown
166 lines
7.0 KiB
Markdown
# CI operations
|
|
|
|
How the pipeline is kept working: the builder image, the secrets it needs, the
|
|
gates that must stay required, and the things that only a human with access to
|
|
the Gitea instance can do.
|
|
|
|
CI is **Gitea Actions** (`.gitea/workflows/`) on `gitea.tourolle.paris`, not
|
|
GitHub.
|
|
|
|
## The workflows
|
|
|
|
| Workflow | Trigger | What it protects |
|
|
|---|---|---|
|
|
| [build-and-test.yml](../../.gitea/workflows/build-and-test.yml) | push/PR to `master` | Frontend + Rust gates, Android compile check, supply chain |
|
|
| [traceability-check.yml](../../.gitea/workflows/traceability-check.yml) | push/PR | Requirement coverage ratchet, dangling IDs |
|
|
| [build-release.yml](../../.gitea/workflows/build-release.yml) | tag `v*` | Builds, signs, publishes, and writes the update manifest |
|
|
| [publish-docs.yml](../../.gitea/workflows/publish-docs.yml) | push to `master` | Docs site on the `gitea-pages` branch |
|
|
|
|
## 🔴 CI installs no system tools
|
|
|
|
Every build, test and packaging **tool** lives in the Docker image the job runs
|
|
in. Never add `apt-get`, `rustup`, `sdkmanager`, or a `curl | tar -xz` of a
|
|
binary to a workflow step.
|
|
|
|
Fetching the project's *own declared dependencies* is not a toolchain install and
|
|
is fine: `bun install`, cargo pulling crates from the lockfile, `cargo deny`
|
|
fetching the RustSec advisory database. The distinction is tool versus data.
|
|
|
|
This rule has been broken twice, both times invisibly until something else
|
|
failed. `publish-docs.yml` downloaded mdBook from GitHub releases into
|
|
`/usr/local/bin` at job time — a hard dependency on GitHub's CDN being up
|
|
whenever docs were published. Both mdBook and the supply-chain tools are in the
|
|
image now.
|
|
|
|
## The builder image
|
|
|
|
`Dockerfile.builder` → `gitea.tourolle.paris/dtourolle/jellytau-builder`.
|
|
It carries: the pinned Rust toolchain plus rustfmt/clippy and the Android,
|
|
Windows-MSVC targets; bun and Node; the Android SDK/NDK and a local Gradle
|
|
distribution; Linux desktop and packaging deps (WebKitGTK, libmpv, rpm, NSIS,
|
|
cargo-xwin); and the tooling — `cargo-deny`, `cargo-cyclonedx`, `mdbook`.
|
|
|
|
Arch packages build in a separate `Dockerfile.arch`, because `makepkg` is
|
|
Arch-specific.
|
|
|
|
### Tags are pinned, and why
|
|
|
|
Workflows name an **immutable dated tag** (`:2026.08`), never `:latest`. While
|
|
every job said `:latest`, rebuilding the image silently changed what every build
|
|
compiled against — including a rebuild of an old release tag, which is the
|
|
opposite of reproducible.
|
|
|
|
`:latest` is still pushed alongside, for local `docker compose` runs and manual
|
|
pulls.
|
|
|
|
Date tags rather than per-commit SHA tags on purpose: the runner shares a 74 GB
|
|
disk with two other projects, and SHA-tagged images accumulated there until it
|
|
filled. Keep a couple of dated tags live and prune the rest.
|
|
|
|
### Changing the image
|
|
|
|
The order matters — CI breaks if the workflow lands before the image exists.
|
|
|
|
```bash
|
|
# 1. Edit Dockerfile.builder. Put new tools in the TRAILING layer: it exists so
|
|
# a tool change is a ~2 min rebuild instead of ~15.
|
|
# 2. Build and push, tagged with the new month:
|
|
./scripts/build-builder-image.sh 2026.09
|
|
# 3. Repoint every workflow at the new tag, in the same commit as whatever
|
|
# needed the new tool:
|
|
sed -i 's|jellytau-builder:2026.08|jellytau-builder:2026.09|g' .gitea/workflows/*.yml
|
|
# 4. Verify the tools are actually in it:
|
|
docker run --rm gitea.tourolle.paris/dtourolle/jellytau-builder:2026.09 \
|
|
-c "cargo deny --version; mdbook --version"
|
|
```
|
|
|
|
🔴 The Rust version is pinned in **two** places that must agree:
|
|
`RUST_VERSION` in `Dockerfile.builder` and `channel` in
|
|
`src-tauri/rust-toolchain.toml`. If they drift, rustup downloads the pinned
|
|
toolchain inside the job — a toolchain install in CI. Bump both, rebuild, push,
|
|
then merge.
|
|
|
|
## Secrets
|
|
|
|
Managed with the `tea` CLI (`tea actions secrets list`) or the repo settings UI.
|
|
|
|
| Secret | Used by | Notes |
|
|
|---|---|---|
|
|
| `ANDROID_KEYSTORE_BASE64` | release | Base64 of the release keystore |
|
|
| `ANDROID_KEYSTORE_PASSWORD` | release | |
|
|
| `ANDROID_KEY_ALIAS` | release | |
|
|
| `ANDROID_KEY_PASSWORD` | release | |
|
|
| `TAURI_SIGNING_PRIVATE_KEY` | release | minisign key for the desktop updater |
|
|
| `TAURI_SIGNING_PRIVATE_KEY_PASSWORD` | release | |
|
|
| `GITEA_TOKEN` | release, docs | PAT; falls back to the auto-provided token |
|
|
|
|
The updater keypair's public half is committed in `src-tauri/tauri.conf.json` —
|
|
that one is meant to be public; it is what clients verify against. The private
|
|
half exists in the Gitea secret and in the maintainer's local `.env` (which is
|
|
gitignored) and at `~/.tauri/jellytau.key`.
|
|
|
|
**Losing the private key means losing the ability to ship updates to installed
|
|
desktop clients**, because they will only accept payloads signed by the key
|
|
matching the public key they were built with. Recovering means generating a new
|
|
pair, shipping a build carrying the new public key, and telling everyone on an
|
|
older build to reinstall by hand. Back it up.
|
|
|
|
## Required status checks
|
|
|
|
Gitea → repo Settings → Branches → protect `master`, requiring:
|
|
|
|
- `Run Tests`
|
|
- `Android Compile Check`
|
|
- `Supply Chain`
|
|
- the traceability job
|
|
|
|
Without branch protection, every gate in this document is advisory: a push
|
|
straight to `master` lands whether or not CI is red. That is the state the repo
|
|
was in for its whole history before this was set up.
|
|
|
|
## The runner
|
|
|
|
One self-hosted runner, one ~74 GB disk shared with two other projects. It fills,
|
|
and when it does the symptoms are misleading: cargo dying mid-link, docker
|
|
refusing to pull, `actions/cache` quietly not saving — anything except an obvious
|
|
out-of-space error. Check the disk first.
|
|
|
|
There is deliberately no scheduled job watching this. On a single-slot runner a
|
|
daily job occupies the slot and pulls the builder image to run `df`, and `df`
|
|
inside a container does not reliably describe the host's disk anyway — it would
|
|
cost real build capacity to report a number that might be wrong. Check it by hand
|
|
on the runner:
|
|
|
|
```bash
|
|
df -h /
|
|
docker system df -v
|
|
```
|
|
|
|
When it does fill:
|
|
|
|
```bash
|
|
docker image prune -a
|
|
docker volume prune -a # the -a matters: without it, NAMED volumes are kept,
|
|
# which is exactly how this filled up unnoticed
|
|
```
|
|
|
|
Never cache `src-tauri/target` — it is ~16 GB, and caching it under several keys
|
|
is what filled the disk at ~1.15 GB/day. The workflows cache only the cargo
|
|
registry index and `.crate` tarballs; cargo re-extracts `registry/src` for free.
|
|
|
|
## Release verification
|
|
|
|
The steps that catch a broken release before users do are in
|
|
[release-checklist.md](../release-checklist.md) — in particular the update path:
|
|
`latest.json` must be live on the `updater` branch, both platform entries must
|
|
carry a non-empty signature, and the previous release should be installed and
|
|
asked to update to the new one.
|
|
|
|
## Bus factor
|
|
|
|
The Gitea instance holds the canonical remote, the signing secrets, the container
|
|
registry and the CI runner. **It is not backed up as part of this repository, and
|
|
nothing in this repository can restore it.** That is the largest single risk to
|
|
the project — larger than any gate in this document — and the backup lives
|
|
outside it.
|