N Nightship

// storage

The Tide delta engine, in depth

How Tide moves only what changed: content-defined chunking, content addressing, signed manifests, and the split that keeps huge builds off your server.

Chunking and content addressing

Each file is split at content-defined boundaries (a FastCDC-style rolling hash), so inserting or removing bytes only re-chunks the affected region instead of shifting every boundary after it — the reason repacked game archives still delta well. Each chunk is hashed with BLAKE3; that hash is the chunk’s address (chunks/<hash> in the bucket). Identical chunks are stored once and reused across files and across builds — global dedup within the bucket. Chunk size is an auto-tuned Small / Medium / Large choice (~1 / ~4 / ~16 MB), locked per channel once builds exist.

Manifests are the build

A manifest is an ordered file tree: each file is a list of chunk hashes and sizes, plus its mode and a whole-file hash. The manifest is the snapshot — promoting a build is just pointing a channel at a manifest. Every manifest is Ed25519-signed, and clients verify the signature before trusting any of the hashes inside it.

Control plane vs data plane

nightshipd is the control plane only: it holds the channel→manifest head pointer, entitlement, and the signing key. All bulk bytes flow from CI to the bucket to the client directly — the server never proxies a chunk. That’s what lets one small box serve very large builds to many testers at once.

Presigned reads, zero client credentials

A sync is a short conversation, then a direct download:

1. The client asks GET /api/tide/sync/{channel}; the server checks entitlement and returns a presigned URL for the manifest plus presigned URLs for the chunks (short-lived; TTL configurable, ~5 min default).
2. The client fetches and verifies the signed manifest, then diffs it against what’s already on disk.
3. It downloads only the missing chunks — to a staging area — then applies them into place atomically per file. A locked target file makes it wait and name the exact path holding things up, rather than failing.

The client holds no standing credentials. It reads only what the server just presigned, and only for as long as those URLs live.

Housekeeping

Because chunks are shared, cleanup is mark-and-sweep, never blind: nightship-tide gc reads every live manifest to compute the set of chunks still in use, then reports what could be deleted (dry-run by default; --delete to actually remove). To check an installed tree against the manifest, nightship-tide verify <channel> <dir> (with --repair) re-hashes on disk and re-pulls only what’s wrong.

← All guides