RollupDB.com

How RollupDB Was Tested With Killed Processes and an Outside Reference

Sat, 17 Oct 2026 00:00:00 +0000

A database that throws data away on purpose had better be right about what it keeps. So RollupDB’s tests ask two questions over and over. Is the answer the same one you’d get from every raw reading? And does a write that was acknowledged survive the process dying, while one that wasn’t leaves no trace?

All the numbers below come from one recorded run on a 4-CPU Linux machine in GitHub’s runners, Go 1.27.2. The script that made them and its full output are in the repository’s test/results folder, so anyone can run it again and compare.

RollupDB Adds SQL, Automatic Rollups and Streams That Can Change

Sat, 17 Oct 2026 00:00:00 +0000

RollupDB 0.4.0 is out. Since the first release three things have been added, one per version: rollups that run on their own, stream definitions you can change after the fact, and a read-only SQL interface over the retained buckets. Each one came out of the same question: what does someone need on day thirty that they didn’t need on day one?

The short answer is that people forget to run advance, they find out their retention was wrong, and they want to ask questions the JSON query wasn’t shaped for. So now rollupdb serve rolls up every stream once a minute, a stream can take a new version with different retention or an added sketch, and every stream is a SQL view.

RollupDB Launches as an Open Source Database for Years of Statistics

Sat, 17 Oct 2026 00:00:00 +0000

RollupDB 0.1 is out, under the Apache License 2.0. It’s an aggregate database in one file: you tell it which statistics matter, and it keeps those for years while it lets the raw events go. Seconds fold into minutes, minutes into hours, hours into days. A query a year later still gets the right count, the right extremes and a percentile with a stated error.

The pitch is simple, and the numbers are what make it worth trying. In the recorded test, five years of readings from two services, 2.8 million of them, took 207 MB as raw JSON lines. The RollupDB file holding all of it was 15.4 MB, with seconds for the last day, minutes for the last day, hours for the last week and a bucket per day going back the full five years.

What an Aggregate Database Keeps When the Raw Events Are Gone

Sat, 17 Oct 2026 00:00:00 +0000

Most databases keep what you put in. An aggregate database doesn’t. It keeps what you’ll ask about later, and only that. You tell it up front: for this stream of latencies, I want the count, the sum, the min and max, the mean, the variance and the 99th percentile, grouped by service. Then it’s free to throw the individual readings away, because it already has what those readings were going to be used for.

Why the Project Is Called RollupDB

Sat, 17 Oct 2026 00:00:00 +0000

A rollup is what monitoring people have called this for decades. You take sixty one-second readings and roll them up into one minute. Then sixty minutes into an hour, and so on, until a year of data fits on a page. Spreadsheets do it with subtotals, OLAP cubes do it along their dimensions, and every metrics system that ever ran out of disk has done it to its old data.

About

Mon, 01 Jan 0001 00:00:00 +0000

RollupDB is an aggregate database in one file. You feed it numeric readings, say which statistics have to survive, and it keeps those statistics in time buckets that get coarser as the data ages. Seconds become minutes, minutes become hours, hours become days. The raw events are never stored, so the file stays small however long the history gets.

It’s open source under the Apache License 2.0 and written in Go. There’s one binary with no server to run and no C compiler needed, plus a Go library and a small HTTP service on the same engine. You can ask it questions as JSON queries or in read-only SQL, and it rolls data up on a schedule while the service runs.

Contact

Mon, 01 Jan 0001 00:00:00 +0000

The project would like to hear from:

Developers trying RollupDB on their own telemetry, with what worked and what didn’t.
Anyone with a workload the measurements should cover: more groups, longer history, a different mix of tiers.
Anyone with a question about the engine, the numbers on this site or how they were measured.

Write to info@rollupdb.com.

If you’re writing about a workload, it helps to say how many readings arrive per second, how many separate series (services, devices, endpoints) there are, how long the history should go back, and which statistics you need at each age.

Download RollupDB

Mon, 01 Jan 0001 00:00:00 +0000

Each archive holds the rollupdb command with its license files and the README. The links always point to the latest release.

System	Download
Linux, x86-64	rollupdb_linux_amd64.tar.gz
Linux, ARM64	rollupdb_linux_arm64.tar.gz
macOS, Apple silicon	rollupdb_darwin_arm64.tar.gz
macOS, Intel	rollupdb_darwin_amd64.tar.gz
Windows, x86-64	rollupdb_windows_amd64.zip

Checksums are in SHA256SUMS. Every release is listed on GitHub.

Install

On Linux or macOS:

tar -xzf rollupdb_linux_amd64.tar.gz
sudo mv rollupdb /usr/local/bin/
rollupdb version

On Windows, unzip the archive and run rollupdb.exe version from the folder.

Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

Five minutes, one binary. You’ll create a database, define a stream of API latencies, load ten thousand readings, ask for their percentiles, roll them up, take a snapshot and ask again in SQL. Every command below was run against the released binary before this page went up.

1. Get the Binary

Download the archive for your system from the download page, unpack it and check it runs:

rollupdb version

2. Say What to Keep

A stream definition is the contract: which dimensions to group by, which statistics to keep, and how long each resolution lives. Save this as latency.json:

Reference

Mon, 01 Jan 0001 00:00:00 +0000

A short map of RollupDB 0.4.1. The full contracts, with every limit and edge case, are in the docs folder of the code repository.

Commands

Command	What it does
`rollupdb version`	Prints the version
`rollupdb init FILE`	Creates a new database; never overwrites
`rollupdb create-stream FILE DEFINITION.json`	Defines a stream
`rollupdb update-stream FILE DEFINITION.json`	Adds the stream’s next version
`rollupdb stream FILE NAME`	Shows a stream’s definition
`rollupdb ingest FILE BATCH.json`	Ingests one batch of up to 10,000 readings
`rollupdb ingest-ndjson FILE STREAM INPUT`	Ingests one reading per line from a file or `-` for stdin, in committed batches (`--batch-size`, default 1000)
`rollupdb query FILE QUERY.json`	Answers a query
`rollupdb sql FILE STATEMENT`	Runs one read-only SQL statement, or `-` to read it from stdin
`rollupdb stats FILE NAME`	Counters, groups, buckets, coverage and tier frontiers
`rollupdb advance FILE [NAME]`	Rolls up buckets that are old enough and expires what’s past retention, for one stream or all of them
`rollupdb snapshot FILE DEST`	Writes a consistent copy to a new file
`rollupdb inspect FILE`	Format version, file ID and stream count
`rollupdb migrate FILE`	Upgrades a file from an older format; back it up first
`rollupdb serve FILE [--listen 127.0.0.1:7070] [--advance-every 1m]`	Runs the local HTTP service and rolls up every stream on a schedule

Stream Definitions

Field	Meaning
`name`, `version`	The stream’s name, and version 1 when it’s created; `update-stream` adds 2, then 3
`effective_at`	Readings before this time are refused
`dimensions`	Up to 16 names to group by, such as `service` or `device`; every reading must carry all of them
`statistics`	Any of `count`, `sum`, `min`, `max`, `avg`, `variance_population`, `variance_sample`
`max_groups`	Cap on distinct dimension combinations; readings for a new one past the cap are refused and counted
`tiers`	From finest to coarsest: `width_seconds`, `retain_seconds` (0 is forever) and `grace_seconds` for late readings
`sketch`	Optional percentiles: `{"algorithm": "kll_float64", "encoding_version": 1, "k": 200, "m": 8}`, with `k` from 64 to 4096

Changing a Stream

update-stream adds the stream’s next version ("version": 2, then 3). A new version may change each tier’s retain_seconds and grace_seconds, max_groups and statistics, and it may add a sketch to a stream that has none. The name, effective_at, dimensions and tier widths stay, since the retained buckets were built with them; anything else is refused.