RollupDB Launches as an Open Source Database for Years of Statistics

RollupDB 0.1 is out, under the Apache License 2.0. It’s an aggregate database in one file: you tell it which statistics matter, and it keeps those for years while it lets the raw events go. Seconds fold into minutes, minutes into hours, hours into days. A query a year later still gets the right count, the right extremes and a percentile with a stated error.

The pitch is simple, and the numbers are what make it worth trying. In the recorded test, five years of readings from two services, 2.8 million of them, took 207 MB as raw JSON lines. The RollupDB file holding all of it was 15.4 MB, with seconds for the last day, minutes for the last day, hours for the last week and a bucket per day going back the full five years.

The more telling number is the other one. Half as many readings, over the same five years and the same two services, made a 15.3 MB file. Doubling the volume grew the file by 0.37%. That’s the whole idea in one line: the size follows how much history you keep and how finely, not how many events came in. Add more services or devices and it grows with them. Eight services instead of two gave 15,888 buckets instead of 3,972 and a 16.3 MB file.

What You Get

• One binary for Linux, macOS and Windows, pure Go, nothing else to install. Download
• A Go library with the same engine, and a small HTTP service on loopback for programs in other languages.
• Exact statistics: count, sum, min, max and mean, plus variance kept in a form that merges correctly.
• Percentiles from a KLL sketch, approximate and labelled that way, with the error model in every answer.
• Snapshots that copy the database to one standalone file you can move anywhere.

What It Won’t Do

RollupDB keeps what you declared and nothing else. A breakdown by a field you didn’t keep can’t be rebuilt later, and a query that asks for detail that has aged out gets told so, with the coverage it does have. There’s no SQL and no clustering in this release, and rollups run when you call advance rather than on a schedule of their own. One process owns a file at a time.

How It Was Checked

Every figure here comes from the recorded run in the code repository: 152 tests, the race detector, and five process-kill tests that each ran three times. The five-year case was checked against a reference computed outside the database, and count, sum, min, max and mean matched it exactly. How RollupDB was tested

The quick start takes five minutes. If it fits your data, or doesn’t, say so.