Pages on RollupDB.com

About

Mon, 01 Jan 0001 00:00:00 +0000

RollupDB is an aggregate database in one file. You feed it numeric readings, say which statistics have to survive, and it keeps those statistics in time buckets that get coarser as the data ages. Seconds become minutes, minutes become hours, hours become days. The raw events are never stored, so the file stays small however long the history gets.

It’s open source under the Apache License 2.0 and written in Go. There’s one binary with no server to run and no C compiler needed, plus a Go library and a small HTTP service on the same engine. You can ask it questions as JSON queries or in read-only SQL, and it rolls data up on a schedule while the service runs.

Contact

Mon, 01 Jan 0001 00:00:00 +0000

The project would like to hear from:

Developers trying RollupDB on their own telemetry, with what worked and what didn’t.
Anyone with a workload the measurements should cover: more groups, longer history, a different mix of tiers.
Anyone with a question about the engine, the numbers on this site or how they were measured.

Write to info@rollupdb.com.

If you’re writing about a workload, it helps to say how many readings arrive per second, how many separate series (services, devices, endpoints) there are, how long the history should go back, and which statistics you need at each age.

Download RollupDB

Mon, 01 Jan 0001 00:00:00 +0000

Each archive holds the rollupdb command with its license files and the README. The links always point to the latest release.

System	Download
Linux, x86-64	rollupdb_linux_amd64.tar.gz
Linux, ARM64	rollupdb_linux_arm64.tar.gz
macOS, Apple silicon	rollupdb_darwin_arm64.tar.gz
macOS, Intel	rollupdb_darwin_amd64.tar.gz
Windows, x86-64	rollupdb_windows_amd64.zip

Checksums are in SHA256SUMS. Every release is listed on GitHub.

Install

On Linux or macOS:

tar -xzf rollupdb_linux_amd64.tar.gz
sudo mv rollupdb /usr/local/bin/
rollupdb version

On Windows, unzip the archive and run rollupdb.exe version from the folder.

Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

Five minutes, one binary. You’ll create a database, define a stream of API latencies, load ten thousand readings, ask for their percentiles, roll them up, take a snapshot and ask again in SQL. Every command below was run against the released binary before this page went up.

1. Get the Binary

Download the archive for your system from the download page, unpack it and check it runs:

rollupdb version

2. Say What to Keep

A stream definition is the contract: which dimensions to group by, which statistics to keep, and how long each resolution lives. Save this as latency.json:

Reference

Mon, 01 Jan 0001 00:00:00 +0000

A short map of RollupDB 0.4.1. The full contracts, with every limit and edge case, are in the docs folder of the code repository.

Commands

Command	What it does
`rollupdb version`	Prints the version
`rollupdb init FILE`	Creates a new database; never overwrites
`rollupdb create-stream FILE DEFINITION.json`	Defines a stream
`rollupdb update-stream FILE DEFINITION.json`	Adds the stream’s next version
`rollupdb stream FILE NAME`	Shows a stream’s definition
`rollupdb ingest FILE BATCH.json`	Ingests one batch of up to 10,000 readings
`rollupdb ingest-ndjson FILE STREAM INPUT`	Ingests one reading per line from a file or `-` for stdin, in committed batches (`--batch-size`, default 1000)
`rollupdb query FILE QUERY.json`	Answers a query
`rollupdb sql FILE STATEMENT`	Runs one read-only SQL statement, or `-` to read it from stdin
`rollupdb stats FILE NAME`	Counters, groups, buckets, coverage and tier frontiers
`rollupdb advance FILE [NAME]`	Rolls up buckets that are old enough and expires what’s past retention, for one stream or all of them
`rollupdb snapshot FILE DEST`	Writes a consistent copy to a new file
`rollupdb inspect FILE`	Format version, file ID and stream count
`rollupdb migrate FILE`	Upgrades a file from an older format; back it up first
`rollupdb serve FILE [--listen 127.0.0.1:7070] [--advance-every 1m]`	Runs the local HTTP service and rolls up every stream on a schedule

Stream Definitions

Field	Meaning
`name`, `version`	The stream’s name, and version 1 when it’s created; `update-stream` adds 2, then 3
`effective_at`	Readings before this time are refused
`dimensions`	Up to 16 names to group by, such as `service` or `device`; every reading must carry all of them
`statistics`	Any of `count`, `sum`, `min`, `max`, `avg`, `variance_population`, `variance_sample`
`max_groups`	Cap on distinct dimension combinations; readings for a new one past the cap are refused and counted
`tiers`	From finest to coarsest: `width_seconds`, `retain_seconds` (0 is forever) and `grace_seconds` for late readings
`sketch`	Optional percentiles: `{"algorithm": "kll_float64", "encoding_version": 1, "k": 200, "m": 8}`, with `k` from 64 to 4096

Changing a Stream

update-stream adds the stream’s next version ("version": 2, then 3). A new version may change each tier’s retain_seconds and grace_seconds, max_groups and statistics, and it may add a sketch to a stream that has none. The name, effective_at, dimensions and tier widths stay, since the retained buckets were built with them; anything else is refused.