How Bootstrapping Was Tested With 400 Killed Applies and Half a Million Checked Values

A tool that writes to customers’ databases on every release makes a few promises, and each one is the kind you only find broken in production. An admin’s edit is never overwritten without a conflict being raised. A change in the seed to a value nobody touched always lands. An apply happens completely or not at all, even if the process dies. Two deploys at once don’t both write. And a value means the same thing to Bootstrapping as it does to SQLite. Each of those got a test built to break it.

Every figure here comes from the files in the repository’s test/results folder, written by one script: sh scripts/record-tests.sh. They were recorded on Linux x86-64 with 2 CPUs, at ten times the size CI runs on every push.

The figures from test/results, drawn together.

Random Histories Against the Rules

The main test plays out the life of one table. Each history runs through 12 seed versions, and every version adds rows, drops rows and changes values at random. Between versions an admin edits values, deletes rows and adds rows of their own. Each apply then settles conflicts one of three ways, at random: keep the local value, take the seed’s, or don’t settle them and let the apply block.

After every apply, the test checks every value against the merge rules. It works those rules out from its own record of what was applied last, never from Bootstrapping’s ledger, so the engine can’t vouch for itself. A value only the admin changed must be the admin’s. A value only the seed changed must be the seed’s. A value both changed must have been reported as a conflict, and must hold whichever side the apply took. Deleted rows must stay deleted, dropped rows must be kept, deleted or retired as the seed file says, and rows the seed never named must be untouched. Then it runs the apply again and checks it finds nothing to do.

Across 1,200 histories that was 14,400 applies and 492,228 values checked. 47,330 local edits were kept and 25,870 seed changes landed. 21,473 conflicts were settled, 11,034 for the seed and 10,439 for the admin, and 3,618 applies were blocked with nothing written, which the test confirms by comparing every table byte for byte. Not one value was overwritten without a conflict.

A test like this is only worth something if it can fail. During development it was run against three deliberately broken versions of the merge: one that overwrites local edits, one that lets conflicts through as updates, one that deletes edited rows. Each failed within its first few histories.

References With Foreign Keys On

Reference data rarely sits in one table. A second random test runs three linked tables with SQLite’s foreign keys switched on: roles that point at a parent role, permissions, and the links between them, each written in the seed by natural key. Its histories add, drop and rename roles, move them between parents, and add and drop permissions and links, while an admin edits the notes on links and deletes some links outright.

After every apply it checks that the apply did exactly what the preview promised, that SQLite finds no broken foreign key, that a renamed role kept its id and its links, that an admin’s note survived unless a conflict was settled for the seed, and that a second apply has nothing to do. Across 600 histories and 6,000 applies, 1,097 renames kept their ids, 2,016 deletes were held back because something still pointed at the row, and 1,819 of 2,078 edited notes were kept, the rest being conflicts settled for the seed.

This test earned its place before release. Its first runs found three ways an apply could fail on a foreign key: a row kept because something pointed at it could itself point at a row due for deletion, rows deleted together could point at each other, and a delete held to the end could still be pointed at by another one held back with it. All three are fixed, and each has its own regression test.

Killed Halfway

An apply runs in one SQLite transaction, which should make it all or nothing. The crash test makes sure. It builds a database with 1,500 seeded rows and some local edits, then applies a new version that takes 1,767 statements, in a separate process, and kills that process with SIGKILL. Half the time it waits until the process reports a chosen statement written inside the transaction. The other half it kills at a random moment, from startup to well after a normal apply would finish, so some kills land during the commit.

It did that 400 times. 269 databases were found exactly as they were before, and 131 exactly as a finished apply leaves them, ledger and change log included. None held anything in between. Every one passed SQLite’s integrity check, and applying again from there produced exactly what a clean apply produces.

Eight Deploys at Once

Two servers starting at the same moment will both try to apply the seed. Bootstrapping takes SQLite’s write lock before it plans, so the second waits, then plans again and finds nothing left to do. The test starts eight processes on one database file and releases them together. In 60 rounds, exactly one process wrote each time, the other seven waited and wrote nothing, and the file matched what a single apply produces.

Values the Way SQLite Sees Them

SQLite converts values as it stores them, by the column’s declared type. The number 1 put in a TEXT column is stored as the text “1”. The text “42” put in an INTEGER column becomes the number 42. If Bootstrapping compared seed values without doing the same, every apply would see changes that aren’t there. So a test stores 4,051 values in columns of 16 declared types, from INTEGER and TEXT to BOOLEAN, DATE and no type at all, and checks the engine’s conversion against what SQLite actually stored. 129,632 checks, and all of them agree.

That test found two real problems before release. SQLite rounds real numbers into text in its own way, so that one conversion is now done by SQLite itself. And the Go driver quietly converts columns declared BOOLEAN, DATE or DATETIME when it reads them, which would have shown false conflicts; Bootstrapping now reads columns in a way the driver leaves alone.

Every Release, Every Machine

The full suite runs on Linux, macOS and Windows on every push, and with Go’s race detector on Linux. Releases go out through a workflow that builds each binary on its own kind of machine and runs a smoke check there before anything is published: the demo with its own checks, a Python script that drives the command through the demo’s story, and eight processes applying at once.