How RollupDB Was Tested With Killed Processes and an Outside Reference

A database that throws data away on purpose had better be right about what it keeps. So RollupDB’s tests ask two questions over and over. Is the answer the same one you’d get from every raw reading? And does a write that was acknowledged survive the process dying, while one that wasn’t leaves no trace?

All the numbers below come from one recorded run on a 4-CPU Linux machine in GitHub’s runners, Go 1.27.2. The script that made them and its full output are in the repository’s test/results folder, so anyone can run it again and compare.

The process-kill test: a child process writes, gets killed outright, the test reopens the file and checks that everything acknowledged is there and nothing from the write in flight is. Zero failures in 15 recorded runs.

Killing the Process

The process-kill tests start a child process, let it write, and kill it outright with no chance to close the file. Then they open the file the way the next program would and check it. There are five of them. One kills after a stream definition is acknowledged. One kills after readings and their percentile sketches are acknowledged. One kills while a write transaction is still open, with buckets, counters and the rollup position half changed. One kills after a finished rollup. The last stops advance just before it commits, kills it, then runs it again to check it does the work exactly once.

Each ran three times in the recorded run, 15 runs in all, with no failures. Everything acknowledged was there, and nothing from an unfinished write was. These tests cover a killed process. They don’t cover a machine losing power, which needs hardware to test properly.

Checking Against an Outside Reference

The five-year test generates its readings outside the database, writes them as JSON lines, feeds them in a day at a time and runs advance daily. Separately, it computes the same statistics straight from the source with exact integer sums and exact rational variance. At the end it compares the two.

It does this three times: 1.4 million readings from two services, then twice the readings, then the original count spread over eight services. In all three, after five years of rollups down to one bucket per day, count, sum, min, max and mean matched the reference exactly. Variance was within a few millionths, the floating-point limit. The 99th percentile landed within about 0.3 percentage points of the true rank, against a stated model of 1.3. A copy of the database, made with snapshot and reopened, matched the reference just as closely, and a stored sketch read back from the copy was identical.

The accuracy test pushes the percentile sketch harder, with 100,000 readings each of uniform, skewed, heavy-tailed, duplicate-heavy and constant data, stored whole and split across seven buckets and merged back. The worst rank error observed was 0.5%, against a model of 1.3%. On constant and duplicate-heavy data, it was zero.

The Rest

The full suite is 152 tests, all passing, plus a pass under Go’s race detector. They cover the arithmetic edges (huge offsets with tiny variance, single readings, overflow), late and out-of-order readings at exact bucket boundaries, injected failures at every step of a rollup, malformed input to the command line and the HTTP service, migration from older file formats, and the clock jumping years ahead. The same suite runs on Linux, macOS and Windows on every push, and a release only goes out after it passes.